[00:00:20] 10Analytics, 10Operations, 10Performance-Team, 10Traffic: Only serve debug HTTP headers when x-wikimedia-debug is present - https://phabricator.wikimedia.org/T210484 (10Krinkle) [00:19:04] 10Analytics, 10Event-Platform, 10Wikimedia-Extension-setup, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Wikimedia-extension-review-queue: Deploy EventStreamConfig extension - https://phabricator.wikimedia.org/T242122 (10Jdforrester-WMF) a:03Jdforrester-WMF [00:19:34] 10Analytics, 10Event-Platform, 10Wikimedia-Extension-setup, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Wikimedia-extension-review-queue: Deploy EventStreamConfig extension - https://phabricator.wikimedia.org/T242122 (10Jdforrester-WMF) p:05Triage→03Medium [06:24:14] moooorning [06:26:47] (03Abandoned) 10Fdans: Fixed - reduce SLA time, include start time in name of cassandra daily coords [analytics/refinery] - 10https://gerrit.wikimedia.org/r/549876 (owner: 10Fdans) [06:29:33] (03Abandoned) 10Fdans: (wip) Change build configuration to put dist assets in subdirectory [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/566768 (owner: 10Fdans) [07:41:23] o/ [07:51:27] Hi folks [13:10:17] milimetric: hello! when you're online, let's talk about the wikistats move? [13:22:46] Thanks a lot elukey for today's rerun :) [13:22:52] np! [13:23:13] I also added some info to https://phabricator.wikimedia.org/T243934 [13:23:25] let me know your thoughts when you have 5 mins :) [13:24:33] elukey: dumb question: would this machine also handle other jobs currently in an-coord1001 like camus, refine? [13:25:02] cause the jobs described in the tickets feel similar [13:29:02] scope to be defined, maybe it could be something that takes a bit of load from the coordinator [13:29:20] or we could simply use the same criteria that we have now with stat1006/7 [13:29:38] if we need xmldumps then we use it, plus also report updater use cases [13:29:55] the vm will likely not be as beefy as a regular stat box [13:30:21] my goal is to only to free stat boxes from our jobs [13:30:26] and separate concernes [13:30:27] beefy-wise it makes sense, functionally a bit less :) I guess it's not really needed :) [13:30:29] *concerns [13:30:52] joal: it is the same problem that we have now with stat1007 - why don't we run camus from there? [13:31:34] or why don't we have the xml dumps on the coordinator? [13:32:12] elukey: I wanted to have xmldumps on the corrd, but we said that adding a host with access to dumps was a bad idea [13:33:05] about reportupdater I have never been part of discussions in the decision of where to run it, but I'd have suggested a different plzace than stat machine, because they are user-sensitive [13:33:26] elukey: about separation of concerns, I definitely take the point [13:37:02] need to run errand for a couple of hours max (doc appointment), but let's keep thinking about this :) [13:37:10] sure elukey :) [13:37:11] joal: feel free to put all your concerns in the task [13:37:12] later! [13:37:27] elukey: not concerned at all - will copy the chan :) [13:37:39] fdans: I’ll be ready in 20 min [13:47:15] Taking a break now to remove a tree-branch from the shelter (that storm has been windy for real in here) - And then I'll need to care the kids as Melissa is out this evening - E-scrum on the way, I'll see y'all later [13:57:08] ok fdans [13:57:11] cave? [13:57:19] milimetric: omw [13:59:54] moins de soux sur mon compte ,mais c'est mieux [13:59:56] oop [14:18:07] (03CR) 10Milimetric: [C: 03+2] Moves all dist assets to ./assets-v2 in the production build [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/570667 (https://phabricator.wikimedia.org/T237752) (owner: 10Fdans) [14:19:18] (03Merged) 10jenkins-bot: Moves all dist assets to ./assets-v2 in the production build [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/570667 (https://phabricator.wikimedia.org/T237752) (owner: 10Fdans) [14:20:37] ottomata: we just merged the wikistats build change, so now when we build /dist will have /dist/index.html and /dist/assets-v2 [14:20:47] you had two different ideas on the symlinks before [14:20:58] cool [14:21:05] two ideas? [14:21:11] I think last I heard you preferred to alias /v2 to the symlinks in / [14:21:20] instead of having 4 symlinks [14:22:07] oh right [14:22:24] i dunno either is fine, the 4 might be better...more explicit? [14:22:35] milimetric: in youur most recent comment on the ticket about v2 vs v3, etc. [14:22:47] were you coming around to just keeping v2/ as canonical urls? [14:22:47] :) [14:23:50] no, I don't think that's a good idea [14:24:40] I was pondering something new there. Think about if we had to "freeze" wikistats 2, we'd have to do something about the two massive clusters that serve it data [14:24:57] either mock them or pre-generate everything, etc. With wikistats 1, we just do nothing [14:25:15] ah [14:25:33] but that's for later, v2 being canonical seems weird [14:26:05] I don't strongly disagree, but I gently think it's not a good idea [14:29:49] i don't care enough to argue :) [14:30:06] ok so milimetric shall we do this thing [14:30:07] ? [14:30:15] have you deplyoed the wikistats change? [14:32:04] not deployed yet, I will do that now [14:32:07] k [14:33:14] milimetric, ottomata : let's rename the v1 index.html to index-v1.html ? [14:33:42] sure [14:34:20] oh yeah, ottomata, hang on [14:34:25] we need to time this a little better [14:34:35] k [14:34:36] fdans: you sending a patch with the links for index-v1.html? [14:34:40] i am at your service! [14:34:48] milimetric: yep [14:35:01] stats.wikimedia.org/index-v1.html [14:35:31] ok ottomata after he does that, I'll build, test, and once we have that change merged, then you can pull, rename index.html, and add the new symlinks [14:35:47] ok [14:35:56] otherwise it'll be broken for a bit. [14:35:58] will prep a puppet patch [14:36:18] and to clean up we should also delete the existing symlink to /v2 at the same time [14:38:21] hm, milimetric just thought of something, [14:38:25] by just using symlinks [14:38:34] we don't get any redirects, which means both URLs look like they are 'canonical' [14:38:56] ottomata: right that's what you said before and I think that's why you wanted to alias instead [14:39:04] or redirect? [14:39:38] right, its been a while since we discussed, if we have the symlinks in / now, an alias from v2/ to / should work, right? [14:40:42] oh no, alias is not reidrect :) [14:40:44] we want redirect [14:40:46] but sure, redirect [14:40:52] redirect, ok [14:42:05] RedirectMatch ^/v2(.*) /$1 [14:42:50] with symlinks in docroot, /index.html -> /dist/index.html, /assets-v2 -> /dist/assets-v2 [14:43:08] Let's do it in two steps then, first symlinks, verify that it works, then redirect [14:44:08] leaving the symlink from /v2 to /dist in the meantime? [14:44:30] oh hm, yeah guess we don't need /dist, right? we can just symlink directly into the git repo clone [14:45:19] index.html -> /srv/src/wikistats-2/dist/index.html, etc. [14:45:22] oh yeah, I thought that's what you meant, the symlinks should be directly to the clone [14:45:25] (the new symlinks) [14:45:26] oh yes, we'd leave the symlink in place for the firs step [14:45:37] the v2 -> .../dist symlink [14:45:45] the old symlink that makes /v2 possible should stay until we get the redirect set up [14:45:47] once everything works with the docroot symlinks, we'll remove it and add tehe redirectr [14:45:49] ya [14:46:00] k [14:52:00] (03PS1) 10Fdans: Add links to v1 wikistats homepage [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/571493 (https://phabricator.wikimedia.org/T237752) [14:53:48] milimetric: ^ [14:58:18] (03CR) 10Milimetric: Add links to v1 wikistats homepage (032 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/571493 (https://phabricator.wikimedia.org/T237752) (owner: 10Fdans) [15:00:18] (03PS2) 10Fdans: Add links to v1 wikistats homepage [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/571493 (https://phabricator.wikimedia.org/T237752) [15:00:38] (03CR) 10Fdans: Add links to v1 wikistats homepage (032 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/571493 (https://phabricator.wikimedia.org/T237752) (owner: 10Fdans) [15:06:43] (03CR) 10Milimetric: [C: 03+2] Add links to v1 wikistats homepage [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/571493 (https://phabricator.wikimedia.org/T237752) (owner: 10Fdans) [15:12:34] (03PS1) 10Milimetric: Release 2.7.0 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/571507 [15:36:00] (03Abandoned) 10Milimetric: Release 2.7.0 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/571507 (owner: 10Milimetric) [15:36:13] abandoned! [15:36:39] it's polluted with some weird semantic stuff, fran will push a clean version while I figure out what's going on [15:36:46] (semantic ui is our css library) [15:38:54] (03PS1) 10Fdans: Release 2.7.0 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/571515 [15:39:02] milimetric: ^ [15:40:47] (03CR) 10Milimetric: [C: 03+2] Release 2.7.0 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/571515 (owner: 10Fdans) [15:42:08] (03Merged) 10jenkins-bot: Release 2.7.0 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/571515 (owner: 10Fdans) [15:44:23] ok ottomata the new release is merged, with links to the original wikistats assumed to be at index-v1.html [15:46:18] milimetric: deployed? [15:46:34] well it auto-deploys when it pulls [15:46:40] but it's merged [15:46:51] ah right ok will run puppet [15:48:01] looks good i see assets-v2 in source [15:48:07] ok milimetric shall we proceed? [15:48:25] yes ottomata [15:49:09] ok https://stats.wikimedia.org/index-v1.html works now [15:49:31] merging symlink creation change [15:52:06] thar she blows [15:52:07] https://stats.wikimedia.org/#/all-projects [15:53:40] intesting, https://issues.apache.org/jira/browse/BIGTOP-3123 shows now that bigtop 1.5 will go to Hadoop 2.10 [15:53:44] rather than jumping to 3 [15:53:53] hm! [15:55:12] I don't mind it, I think that there are some packaging issues with Hadoop 3 stuff, better to be careful :) [15:55:23] Hadoop 2.10 + Hive 2.3 would be nice though! [15:56:13] we could do Stretch + BigTop 1.4 + hadoop 2.8.5, then Stretch + BigTop 1.5 + hadoop 2.10 and finally buster [15:56:37] and then hadoop 3 when bigtop is ready [15:56:47] +1 [15:56:51] why not skip bigtop 1.4? [15:57:15] 1.5 is still no released, plus jumping from hdfs 2.6 to 2.8 seems less risky [15:57:20] maybe [15:57:23] not sure :) [15:57:50] i'd hope that any jumps within same major version would be ok [15:58:17] yes I'd hope the same [15:58:23] milimetric: i just quick live tested the redirect change, it works. you ok if I merge that one and permanantly redirect /v2 -> / ? [15:58:39] ottomata: looks good [15:58:44] testing the main site, all looks good [15:58:46] thank you! [16:00:12] nuria: you agree to keeping /v2 urls as canonical?!!! [16:00:31] as in, stats.wikimedia.org will redirect to stats.wikimedia.org/v2 ? [16:01:21] milimetric: I was trying to say need to unbundle all "v2" urls to be redirected to remove the v2 [16:01:42] nuria: oh ok, that's the opposite :) [16:01:55] oook [16:01:57] milimetric:sorry, me no typing well [16:02:01] https://stats.wikimedia.org/v2/ now redirects to /! [16:02:06] all done i thnk! [16:02:13] all good, we done, yep [16:02:35] wikistats 2 is live, and there's no cake here for me to stuff into my face to adequately show how excited I am :) [16:02:51] * milimetric stuffs a GIANT SLICE of virtual cake into his face [16:03:03] <---- <) [16:04:18] milimetric: you have to wear a special hat [16:04:36] * milimetric puts on a Coco Channel hat [16:15:18] milimetric: good thing to announce in the wikipedia weekly group! [16:15:26] so people will be aware etc... [16:16:15] yep [16:19:57] ottomata: milimetric thank you both for doing this [16:20:10] :D [16:21:08] links in v1 look good [16:24:12] ok, so let's remove the migration banner from the v1 site as well [16:24:26] ottomata: can I get access to write to the site as well? [16:24:37] *to the index-v1.html? [16:24:46] actually how should I edit that [16:26:08] milimetric: you should have it [16:26:12] it is editable by folks in wikidev group [16:26:22] sweet, on thorium directly, right? [16:26:23] thorium [16:26:25] yup [16:26:29] /srv/stats.wikimedia.org/htdocs/index-v1.html [16:28:57] milimetric: if you have time, shall we also remove the old stuff? https://phabricator.wikimedia.org/T237752#5667542 [16:32:01] Ah i was just asking dan about that stuff in a PM [16:32:05] was going to bring it up in standup [16:32:11] cool so we can just drop all that? [16:32:27] IIRC we needed to triple check with some people that the data was not needed [16:32:38] but then I didn't receive any update :) [16:34:51] elukey: just drop it, we have to archive the data as part of the ez directory cleanup anyway [16:34:59] and that's blocked on the pagecounts-ez migration [16:35:33] but this config is almost surely not used by anyone. And if anyone is using it, I'd love to talk to them, so let's drop it and wait and see [16:37:06] milimetric: drop the config, not the directory on the host right? Or also the data? [16:37:11] (triple checking) [16:54:31] elukey: just the config, the directories are part of the confusing legacy wikistats 1 structure that we're talking about in the ez cleanup task [16:55:19] everything's connected there, so I think our latest thoughts were to archive everything to hdfs then prune one by one from other places, like the various backups/copies on thorium and stat1007 [17:29:34] 10Analytics: Run a script to check REFINE_FAILED flags daily - https://phabricator.wikimedia.org/T240230 (10elukey) This happened today T244765, hopefully the script will prevent it :) [17:29:44] 10Analytics, 10User-Elukey: Run a script to check REFINE_FAILED flags daily - https://phabricator.wikimedia.org/T240230 (10elukey) [17:32:36] ottomata: I can try to work on --^ so I'll learn a bit of scala, starting from your poc [17:32:47] should be an easy one [17:32:52] (famous last words) [17:33:00] nice! i was wondering if that would be automated :) [17:59:26] a-team: Hey everyone, here's the design doc I created for the AQS Geoeditors API I'm working on: https://docs.google.com/document/d/1D-v2vTtFt94xZ9HVSky7BKpzF4H2LZ2yC5H9KR_rgKI/edit . If you could take a look and provide any feedback you have, that would be great! [17:59:35] fdans: so next thing we should work on - mediarequests. I'm caught up, want to talk tomorrow morning or should I do anything tonight? [17:59:48] will look after lunch lexnasser [18:02:13] 10Analytics, 10Operations, 10serviceops, 10vm-requests, 10User-Elukey: Create a replacement for kraz.wikimedia.org - https://phabricator.wikimedia.org/T244719 (10elukey) >>! In T244719#5872221, @Ottomata wrote: > I've never understood why we do this. Wouldn't it be better to create the VM in the private... [18:02:25] milimetric: yea lets talk tomorrow morning [18:08:09] * elukey off! [18:13:04] fun find: pyarrow, bundled with pyspark, can talk direct to hdfs via pyarrow.hdfs.HadoopFileSystem() (using libhdfs) in python3 [18:26:43] 10Analytics, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Patch-For-Review: Refining is failing to refine centranoticeimpression events - https://phabricator.wikimedia.org/T244771 (10Nuria) Ping @jkumalah so she knows that this is going on [18:29:31] 10Analytics, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Patch-For-Review: Refining is failing to refine centranoticeimpression events - https://phabricator.wikimedia.org/T244771 (10jkumalah) thanks for the ping @Nuria . Will follow-up with @AndyRussG for additional insights. [18:29:58] 10Analytics, 10Operations, 10serviceops, 10vm-requests, 10User-Elukey: Create a replacement for kraz.wikimedia.org - https://phabricator.wikimedia.org/T244719 (10Ottomata) Don't we want to make the service HA? [18:30:13] 10Analytics, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Patch-For-Review: Refining is failing to refine centranoticeimpression events - https://phabricator.wikimedia.org/T244771 (10Nuria) Also, in order to do the archiving requested here by Advancement/FR: {T161656} of this years banner... [18:31:32] 10Analytics, 10Analytics-Wikistats: Mediawiki History Dumps - Possible parsing issue - https://phabricator.wikimedia.org/T244807 (10leila) @JAllemandou thanks for checking. I'll resolve the task then. (I don't know if the pipe is causing this flip in text.) [18:31:42] 10Analytics, 10Analytics-Wikistats: Mediawiki History Dumps - Possible parsing issue - https://phabricator.wikimedia.org/T244807 (10leila) 05Open→03Resolved [19:24:40] Hey all, just wondering if there's a way to speed along https://phabricator.wikimedia.org/T244773 ? [19:24:50] (getting a kerberos password) [19:28:41] 10Analytics: Create a Kerberos access for sguebo - https://phabricator.wikimedia.org/T244913 (10sguebo_WMF) [19:28:54] ^ related lol [19:31:16] 10Analytics: Create a Kerberos identity for foks - https://phabricator.wikimedia.org/T244773 (10Nuria) @jrbs Do you have a current user to access the analytics cluster? [19:36:08] 10Analytics: Create a Kerberos identity for foks - https://phabricator.wikimedia.org/T244773 (10jrbs) >>! In T244773#5873923, @Nuria wrote: > @jrbs Do you have a current user to access the analytics cluster? I currently use stat1005 with foks, yes. I've used it before just never with Kerberos (it's pretty rare... [19:45:42] 10Analytics, 10Operations, 10serviceops, 10vm-requests, and 2 others: Create a replacement for kraz.wikimedia.org - https://phabricator.wikimedia.org/T244719 (10elukey) >>! In T244719#5873618, @Ottomata wrote: > Don't we want to make the service HA? I don't think it is a goal for now, we'd need a replacem... [19:52:13] ottomata: my impression from Faidon about --^ was that he wanted a replica of kraz in ganeti (in codfw) with a public IP to test the new software etc.. [19:52:44] so the VM might end up into some service in k8s after the test, I am not sure [19:53:07] I am only trying to enable proper testing, no idea about long term deployment [19:53:27] (afk again! :) [19:55:04] Thanks lexnasser for your design doc - I added some comments :) [20:06:05] 10Analytics, 10Analytics-Wikistats: [Wikistats v2] Default selection for (active) editors is confusing for inexperienced users - https://phabricator.wikimedia.org/T213800 (10Tgr) Yeah, including IP edits in the editor / active editor count is horribly unintuitive. I had to redo the analysis in {T209224} after... [20:06:36] 10Analytics, 10Operations, 10serviceops, 10vm-requests, and 2 others: Create a replacement for kraz.wikimedia.org - https://phabricator.wikimedia.org/T244719 (10Ottomata) I thought one of the issues with the current IRC daemon was that it can't ever be taken offline. Without an HA or at least auto-failove... [20:08:12] elukey: re REFINE_FAILED script, ok! [20:08:36] re rc irc, commented on task, but ok! [20:15:34] 10Analytics: Create a Kerberos identity for foks - https://phabricator.wikimedia.org/T244773 (10Nuria) ping @elukey for kerberos user [20:20:33] 10Analytics, 10User-Elukey: Redesign architecture of irc-recentchanges on top of Kafka - https://phabricator.wikimedia.org/T234234 (10Krinkle) >>! In T234234#5864987, @Ottomata wrote: > [[ https://phabricator.wikimedia.org/T242712#5864926 | EventStreams or Kafka ]]!? > > Hm, the proposal in this ticket isn't... [20:20:39] elukey: not sure if you are off for the day, if you arent would like to ask you about good ol' GSS initiate failed happenign for refine sanitize [20:32:08] 10Analytics, 10User-Elukey: Redesign architecture of irc-recentchanges on top of Kafka - https://phabricator.wikimedia.org/T234234 (10Krinkle) >>! In T234234#5874081, @Krinkle wrote: >>>! In T234234#5864987, @Ottomata wrote: >> [[ https://phabricator.wikimedia.org/T242712#5864926 | EventStreams or Kafka ]]!? >... [20:34:57] 10Analytics, 10User-Elukey: Redesign architecture of irc-recentchanges on top of Kafka - https://phabricator.wikimedia.org/T234234 (10Ottomata) > It seems natural to me that it would use Kafka when actually deployed +1 [21:07:08] joal: talk design document tomorrow morning (the one with CPT)? [21:07:17] sure milimetric [21:07:22] I'm doing a schema [21:07:31] I know you said you'd work late but figured this was too late :) [21:08:52] milimetric: we can talk now if you want :) [21:09:22] I'll be in the cave until you fall asleep :) [21:09:34] huhu [21:13:43] 10Analytics, 10Growth-Team, 10Product-Analytics: Hash edit session ID in EditAttemptStep and VisualEditorFeatureUse whitelisting - https://phabricator.wikimedia.org/T244931 (10nettrom_WMF) [21:14:21] 10Analytics, 10Growth-Team, 10Product-Analytics: Hash edit session ID in EditAttemptStep and VisualEditorFeatureUse whitelisting - https://phabricator.wikimedia.org/T244931 (10nettrom_WMF) The two listed schemas are the ones I know about that uses the editing session ID. If there are others, please do add them! [21:22:31] ottomata: o/ - yes I think I know what's happening with refine - sanitize_etc.. uses profile::analytics::refinery::job::refine_job but not use_kerberos => true [21:22:38] so the keytab is not passed along [21:22:45] and it fails in that way for ALTER etc.. [21:24:39] there is still an outage ongoing so I will not send the patch, but will do it tomorrow mornign first thing! [21:38:18] ok luca ty! [23:32:55] (03CR) 10Nuria: Add spark code for wikidata json dumps parsing (033 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/346726 (https://phabricator.wikimedia.org/T209655) (owner: 10Joal) [23:40:00] (03CR) 10Nuria: [C: 04-1] "Couple nits about docs but I am missing where is one of the values setup" (033 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/569836 (https://phabricator.wikimedia.org/T209655) (owner: 10Joal) [23:43:21] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Enable shell access to presto from jupyter/stats machines - https://phabricator.wikimedia.org/T243312 (10Nuria) is prestodb installed via debian or something it needs to be installed via pip in a virtaul env? [23:46:48] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Enable shell access to presto from jupyter/stats machines - https://phabricator.wikimedia.org/T243312 (10Nuria) CLI works great and it is SO very fast!