[00:06:23] 10Analytics, 10Discovery-Analysis, 10Product-Analytics, 10Reading-analysis, 10Patch-For-Review: Productionize per-country daily & monthly active app user stats - https://phabricator.wikimedia.org/T186828 (10chelsyx) I ran the two queries of option 1 ("new") and 4 ("old") in T186828#4505784 for both iOS a... [05:28:43] hello :) [05:54:00] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10Patch-For-Review: Review analytics-in4/6 rules on cr1/cr2 eqiad - https://phabricator.wikimedia.org/T198623 (10elukey) From my tcpdumps it seems that no more https calls are made via ipv6 without going through the proxy. @ayounsi, we can proceed... [06:26:13] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Upgrade Archiva (meitnerium) to Debian Stretch - https://phabricator.wikimedia.org/T192639 (10elukey) Very interesting chat happened on IRC between @Gehel and @Ottomata about the current repository structure in Archiva (pros/cons/shortcomi... [06:26:17] joal: morninggggg [06:26:36] I added a past conversation about archiva in --^ [06:26:49] about mirrored releases etc.. [07:26:17] Morning elukey :) [07:26:49] hello :) [07:29:20] indeed elukey - interesting discussion [07:30:01] joal: one thing that I realized though is that reaching consensus will likely require a lot of time [07:30:08] so it might be wise to split the two tasks [07:30:22] I have rsynced the archiva repos to archiva1001 [07:30:30] now it is only a matter of configuring them via UI [07:30:35] and we should be good to test [07:30:39] elukey: my understanding of the talk is that consensus is actually already there, but understanding is not shared [07:30:51] maybe I'm misunderstanding though [07:30:59] well not really afaics, what about the mirrored repo? [07:31:11] it is not clear if the current state of things is good or not [07:31:39] same thing for reseases (for example, we have in there gerrit jars, that are not really produced by us afaik) [07:31:52] elukey: AFAIK mirrored is never used in a manual-upload way (at least by analytics) - so as Andrew said for us it's a "proxy-only" repo [07:32:15] ah yes for us [07:32:29] elukey: about other teams, I can't speak :) [07:32:45] yeah this is the issue :D [07:32:55] as part of this task I'd like to document that [07:33:02] in the archiva's wikitech page [07:33:06] sounds good [07:33:08] and then send an email etc.. [07:33:13] and very much needed :) [07:33:23] but a more carefuly review of those repos needs to happen [07:33:45] for example, as geh*el was saying, it would be awesome to have a review process before uploading jars [07:33:51] I mean third party jars [07:34:12] WHile I uhnderstand, [07:34:35] it also means slower dev-cycles - I'm not againstn [07:39:48] elukey: on a completely different matter - I'd like to deploy the cluster today or tomorrow, with some big changes [07:40:04] elukey: Will write a plan and ask for a review :) [07:41:59] joal: sure! whenever you have time archiva-new should be ready to test :) [07:42:16] * gehel jumps in the middle of that conversation... [07:42:25] elukey: the hdfs mount on stat1005 seems broken [07:42:38] there's an Icinga failure for the disk space check [07:42:43] also, I'd need to reboot the whole analytics cluster for kernel/etc.. security upgrades, so I'll start probably tomorrow to avoid any issue [07:42:46] the archiva "mirror" repo isn't actually a proxy (as far as I understand the Archiva configuration) [07:42:47] moritzm: ack! [07:43:07] so it looks very much like that "mirror" repo is a manual mirroring of external dependencies [07:43:10] and debdeploy also threw me an error that stat() failed for fuse.fuse_dfs [07:43:17] ack [07:43:47] gehel: hello :) So for the moment I rsynced (and configured) the repos as they are on archiva.w.o, to separate the two tasks (upgrade archiva vs reshaping its repos) [07:43:52] Hi gehel [07:44:07] elukey: good idea! no need to block the upgrade! [07:44:30] gehel: AFAIK the analytics usage of the mirrored repo is to "WMF-local" store the dependencies needed to build our projets [07:45:01] joal: dependencies that are not present in Maven Central or other public maven repositories? [07:45:02] moritzm: yeah fuse_dfs is really flaky :( [07:45:38] gehel: any dependency that we doenload from central or other external repo if I understand correctly [07:46:33] strange... [07:47:05] gehel: However, I am completely unaware of how archiva has been configured - I'm talking from a usage PoV only [07:47:13] gehel: And, talking for analytics only, too [07:47:43] as far as I can see, we have repos setup to automagically proxy central, cloudera and spark, so we should not need to upload anything manually for dependencies that are available in one of those repo [07:48:20] gehel: I have never uploaded any jars onto "mirror" (or it was a mistake) [07:49:06] ok, make sense then. Not sure what "mirrored" is supposed to be. And I'm pretty sure it is full of jars that are available on Central [07:50:31] gehel: For sure! I wasn't here when the setup was devised, but my understanding is that archiva should allow us to build our projects event if central is unavailable [07:50:45] And actually, our projects normally don't rely on central at all [07:50:58] They rely on archiva, which in turn mirror central [07:51:22] proxy central more than mirror central, but yep, that's my understanding as well [07:51:45] (there is really no need for us to mirror central completely, since we probably use < 1% of what's there) [07:51:48] \o/ ! Up to some common understanding :) [07:52:25] I think that you, elukey and Andrew (and me) all share the same understanding. [07:52:39] It's just the reality of what is in Archiva that disagree ! [07:52:40] gehel: My unstanding is that jars are downloaded onto archiva only when they are needed for a project, meaning we don't mirror central fully, but only for the needed jars [07:52:50] joal: correct! [07:52:59] gehel: ok great [07:53:14] Let's just forget about reality :) Our made up world is so much better! [07:53:29] gehel: MEEEH? Archiva, tell me, in which reality do you live? [07:53:51] gehel: (to understand) what are your concerns now with mirrored? The fact that we mirror vs proxying only central (and others) ? [07:54:07] or that we have proof of people uploading to mirror jars? [07:54:19] gehel: in order to move archiva's reality to ours, is it very complicated or is it mostly about some config to be updated? [07:55:02] Honestly, this is mostly about my OCDs, there is no immediate problem in having crap in Archiva [07:55:35] We're wasting disk space by duplicating a bunch of crap (I don't think archiva deduplicates storage). [07:56:11] We are in a situation where you are required to read docs to understand what should be going on, since looking at what is actually in archiva will get you the wrong idea. [07:56:37] And we are having people uploading crap in the wrong repo, probably making their own life more complex than it should be. [07:56:55] * gehel is probably one of the people who uploaded crap [07:57:44] gehel: Ah - As I said to elukey, I have no clue of how other teams use Archiva :( [07:58:26] I have some idea of how search use it, and the short answer is that we mostly don't :) [08:01:32] ahhh it is lovely when you start a task to upgrade something and then you open the Pandora's box :D [08:02:24] jokes aside, I'll make sure to collect all these thoughs in a new task to establish if/how to reshape archiva [08:02:57] atm archiva-deploy (and the new LDAP auth model) allows people to be global repo managers [08:03:07] thanks elukey :) [08:03:15] that doesn't seem to be the good perm from what I can read [08:03:28] so another thing that would need a more granular approach [08:03:33] but, one step at the time :) [08:03:53] gehel,joal - if you want (whenever you have time) to test archiva-new it would be great :) [08:04:46] elukey: adding LDAP is great! And I'm sure it will help bring some order in the chaos! [08:04:49] Thanks ! [08:05:20] gehel: Will do today, but after my bunch of other things :) [08:07:38] 10Analytics: Scripts modified to allow a https proxy on stat1005 - https://phabricator.wikimedia.org/T201134 (10elukey) 05Open>03Resolved From my point of view all the https connections are now using the http proxy, didn't see any issue so far. I am going to close this task, please re-open if anything is not... [08:07:44] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10Patch-For-Review: Review analytics-in4/6 rules on cr1/cr2 eqiad - https://phabricator.wikimedia.org/T198623 (10elukey) [08:40:11] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Upgrade Archiva (meitnerium) to Debian Stretch - https://phabricator.wikimedia.org/T192639 (10elukey) I've rsynced /var/lib/archiva/repositories to archiva1001 and configured them, so archiva-new.wikimedia.org should be ready for the first... [08:56:51] 10Analytics, 10User-Elukey: Add a safe failover for analytics1003 - https://phabricator.wikimedia.org/T198093 (10elukey) 05Open>03Resolved [09:02:35] joal: https://wikitech.wikimedia.org/wiki/Archiva#Uploading_dependency_artifacts [09:02:41] "If your project has a dependency that is not already in the wikimedia.mirrored repository, you may want to add this manually" [09:02:50] meh [09:02:55] * elukey plays sad_trombone.waw [09:03:01] *wav [09:11:02] created https://wikitech.wikimedia.org/wiki/Archiva#Repositories [09:13:01] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Upgrade Archiva (meitnerium) to Debian Stretch - https://phabricator.wikimedia.org/T192639 (10elukey) Started a draft of https://wikitech.wikimedia.org/wiki/Archiva#Repositories [09:16:29] 10Analytics, 10Patch-For-Review: Review Bacula home backups set for stat100[56] - https://phabricator.wikimedia.org/T201165 (10elukey) The size of the home directories might not be eligible for backup: ``` elukey@stat1004:/srv/home/elukey$ sudo du -hs /srv/home/ 959G /srv/home/ elukey@stat1005:/srv/home/e... [09:40:02] (03CR) 10Joal: [C: 031] "Looks correct to me as well. Let fix merge conflict and merge :)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/453010 (https://phabricator.wikimedia.org/T198600) (owner: 10Fdans) [09:55:05] (03CR) 10Joal: [C: 031] "With https://gerrit.wikimedia.org/r/c/analytics/refinery/+/448504 having been merge, let's update basename to YYYY_MM instead of YYYY-MM" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/448551 (https://phabricator.wikimedia.org/T197889) (owner: 10Fdans) [09:58:36] (03PS3) 10Joal: Update mediawiki-user-history empty-registration [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/454338 (https://phabricator.wikimedia.org/T201620) [10:08:55] elukey: for when you have a minute: https://gist.github.com/jobar/fb88bd24beb0ef5a6eaa50771331ac82 [10:11:41] joal: just read it, but not sure how to read the "To Be updated/reviewed" [10:11:46] are those show stoppers? [10:12:16] elukey: I think those tasks are close enough to completion and I'd like to wait for them [10:13:15] At least the first 2 - The last 2 are about cleanup, and fdans is on them, but I think we are super close as well on them [10:14:16] ah okk! Are we planning to deploy in the afternoon? [10:14:54] elukey: I'd say tomorrow, waiting for confirmation on reviews etc [10:16:09] okok, so can I start the round of reboots today? [10:16:46] also if we could use archiva-new it would be awesome :) [10:17:08] joal: not sure what merge conflict was there to solve, I merged https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/448504/ no problem :) [10:17:41] fdans: wrong link I presume :) [10:18:35] elukey: using archiva-new can be done, but will make the whole deploy somehow more complicated: manual upload of jars, and manual patch of artifacts/ folder in refinery [10:18:40] elukey: doable though [10:20:31] joal: we could do some testing beforehand, and I should be able to flip archiva.wikimedia.org to archiva1001 to alleviate the pain [10:20:47] Ah ! [10:21:06] hm elukey - I wonder about jenkins deploy user [10:21:18] It would actually be an interesting test [10:21:25] it is a matter of creating archiva-ci to archiva-new, will do now [10:25:11] done [10:25:36] so it should only be a matter of pointing archiva.w.o to the new host after some tests [10:25:52] awesome elukey [10:26:12] elukey: I'm not sure about what tests I should do though :( [10:26:36] elukey: Will try to build out of archiva-new (jars retrieval) [10:26:39] joal: I am a bit ignorant but maybe trigger a build of refinery downloading artifacts etc.. [10:26:44] yeah exactly [10:27:13] elukey: about upload, I can try to upload a test jar if you want [10:27:25] a fake one [10:29:47] sure sure feel free to do all the tests you want :) (thanks!) [10:29:57] going afk for lunch + errand, ttl! [10:30:05] bye :) [10:30:22] (I'll also try to RTFM myself and do some tests, but yours are surely more reliable :P) [10:31:24] elukey: not sure at all ;) [12:11:49] 10Analytics-Tech-community-metrics, 10Developer-Advocacy (Oct-Dec 2018), 10Upstream: Inconsistent numbers between number widgets and list widgets; some accounts counted twice - https://phabricator.wikimedia.org/T184741 (10Aklapper) This hopefully should get fixed with the Kibiter 6 upgrade in Sep/Oct 2018. [12:13:02] 10Analytics-Tech-community-metrics, 10Developer-Advocacy (Jul-Sep 2018), 10Upstream: Affiliations/enrollments not always correctly synced between user data in database and frontend indices - https://phabricator.wikimedia.org/T191779 (10Aklapper) This is being worked on: "A fix was just included in the latest... [12:20:14] 10Analytics-Tech-community-metrics, 10Developer-Advocacy (Oct-Dec 2018): Advertise wikimedia.biterg.io more widely in the Wikimedia community - https://phabricator.wikimedia.org/T179820 (10Aklapper) Moving from Q3 to Q4 as per status updates on subtasks [12:22:28] 10Analytics-Tech-community-metrics, 10Upstream: "Wiki Editions" should be "Wiki edits" - https://phabricator.wikimedia.org/T164935 (10Aklapper) https://github.com/chaoss/grimoirelab-sigils/pull/232 got merged; waiting for deployment in production [13:01:40] 10Analytics, 10Patch-For-Review: Review Bacula home backups set for stat100[56] - https://phabricator.wikimedia.org/T201165 (10akosiaris) Hm 6TB ? Yes, although doing so currently might be a bit too much. We are however refreshing the infrastructure for both codfw/eqiad and the new one definitely has the free... [13:08:56] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Turn off old geowiki jobs - https://phabricator.wikimedia.org/T190059 (10fdans) Verified that Geowiki Legacy on superset has data until the end of March 2018. Geoeditors Monthly has data since January 2018, so the geowiki data on stat1006 should be ok to d... [13:47:11] 10Analytics, 10Operations, 10Documentation: Remove data from Hadoop's HDFS as part of the user offboard workflow - https://phabricator.wikimedia.org/T200312 (10elukey) The users might leave PII data in the following places: * /home/$USER dir on the stat boxes * /user/$USER dir on HDFS * Hive databases on HDFS [13:55:02] 10Analytics, 10Product-Analytics, 10Reading-analysis: Assess impact of ua-parser update on core metrics - https://phabricator.wikimedia.org/T193578 (10fdans) I do apologize for the delay on this. I've run the same study on bots as in https://phabricator.wikimedia.org/T193578#4196915 but adding `is_pageview=t... [13:57:37] (03PS7) 10Fdans: Add druid snapshot deletion script [analytics/refinery] - 10https://gerrit.wikimedia.org/r/448551 (https://phabricator.wikimedia.org/T197889) [13:59:07] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Drop mediawiki history old snapshots from druid public cluster - https://phabricator.wikimedia.org/T197889 (10fdans) [13:59:22] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Drop mediawiki history old snapshots from druid public cluster - https://phabricator.wikimedia.org/T197889 (10fdans) [14:00:00] joal: just changed this to have an underscore, ok to merge? https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/448551/ [14:05:18] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Turn off old geowiki jobs - https://phabricator.wikimedia.org/T190059 (10fdans) @Nuria Correction: I meant the data in Thorium. The data in stat1006 is already removed like I said in my comment a couple weeks ago. [14:15:27] hey fdans - sorry, missed your ping [14:15:32] fdans: yes, ok to merge :) [14:16:13] (03CR) 10Fdans: [V: 032 C: 032] Add druid snapshot deletion script [analytics/refinery] - 10https://gerrit.wikimedia.org/r/448551 (https://phabricator.wikimedia.org/T197889) (owner: 10Fdans) [14:17:00] managed to build refinery source with archiva-new [14:17:06] (03PS1) 10Milimetric: Add new non-wiki privacy policy [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/455567 (https://phabricator.wikimedia.org/T202792) [14:19:32] elukey: heya [14:19:38] elukey: I got an error :( [14:19:44] elukey: have you changed something? [14:20:07] Oh ! Looks like you have elukey - success for me as well :) [14:20:29] (03PS8) 10Fdans: Add druid snapshot deletion script [analytics/refinery] - 10https://gerrit.wikimedia.org/r/448551 (https://phabricator.wikimedia.org/T197889) [14:21:10] (03CR) 10Fdans: [V: 032 C: 032] Add druid snapshot deletion script [analytics/refinery] - 10https://gerrit.wikimedia.org/r/448551 (https://phabricator.wikimedia.org/T197889) (owner: 10Fdans) [14:22:20] (03PS1) 10Milimetric: Add non-wiki privacy policy [analytics/analytics.wikimedia.org] - 10https://gerrit.wikimedia.org/r/455569 (https://phabricator.wikimedia.org/T202792) [14:22:35] joal: gooood! [14:23:02] There was an error due to guest user not being able to download artifacts from the repos (basically all unauthenticated users) [14:23:04] elukey: I had a failure trying to download artifacts in previous try [14:23:14] ah yes that might have been it then [14:23:28] it took me a bit to figure out what the hell was happening :D [14:23:38] cool :) [14:23:56] joal elukey: will post a patch now to add the new druid script to puppet :) [14:24:05] \o/ ! [14:24:15] fdans: I plan to deploy the cluster tomorrow (ifb [14:24:34] (if milimetric agrees on merging the oozie + AQS code) [14:25:38] I'll take a look in a bit at the patches [14:25:51] Fdv-cupertino-090190 [14:27:53] many thanks milimetric [14:28:59] (03PS1) 10Milimetric: Add non-wiki privacy policy [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/455571 (https://phabricator.wikimedia.org/T202792) [14:32:10] (03CR) 10Milimetric: [V: 032 C: 032] Update Wikistats2 top and per-editors/edited-pages [analytics/aqs] - 10https://gerrit.wikimedia.org/r/454243 (https://phabricator.wikimedia.org/T201617) (owner: 10Joal) [14:32:59] joal: you can merge this whenever you like: https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/454242/ [14:33:29] (03CR) 10Milimetric: [C: 032] Add MediawikiHistoryChecker spark job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/439869 (https://phabricator.wikimedia.org/T192481) (owner: 10Joal) [14:34:12] (03CR) 10Milimetric: "Any thoughts on my comment here?" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/445373 (https://phabricator.wikimedia.org/T192483) (owner: 10Joal) [14:34:42] joal: same with this, merge whenever you want: https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/440005/ [14:35:04] (03CR) 10Milimetric: [C: 032] Update MediawikiHistoryChecker adding reduced [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/441378 (https://phabricator.wikimedia.org/T192481) (owner: 10Joal) [14:35:26] ok, I +2-ed everything but the oozie job, which I have a pending question about ^ [14:36:07] milimetric: sure [14:36:28] milimetric: let's also discuss the by-IP endpoint post-standup if you want [14:37:06] milimetric: the last one if you may: https://gerrit.wikimedia.org/r/c/analytics/refinery/source/+/454338 [14:39:42] joal is there a way for me to get the public druid host name from puppet, or should I just hardcode it in purge.pp? [14:40:10] you should use hiera [14:40:28] but only if you are working on a profile [14:40:46] otherwise make it a param of the class, and then pass it from the correspondent profile [14:42:52] (03CR) 10Milimetric: [C: 032] Update mediawiki-user-history empty-registration (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/454338 (https://phabricator.wikimedia.org/T201620) (owner: 10Joal) [14:43:14] joal: +2 but see comment to update TODO [14:43:21] (03Merged) 10jenkins-bot: Add MediawikiHistoryChecker spark job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/439869 (https://phabricator.wikimedia.org/T192481) (owner: 10Joal) [14:44:18] 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review: Move internal sites hosted on thorium to ganeti instance(s) - https://phabricator.wikimedia.org/T202011 (10Ottomata) > I'm not sure if it's feasible to keep the rest of Cloudera and only install Hue from the upstream sources? This might be... [14:44:18] thank youuuu elukey [14:44:29] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Turn off old geowiki jobs - https://phabricator.wikimedia.org/T190059 (10Nuria) ok, let's delete old data then. [14:45:11] hola [14:45:25] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Upgrade Archiva (meitnerium) to Debian Stretch - https://phabricator.wikimedia.org/T192639 (10elukey) Result for `mvn package` set with archiva-new.wikimedia.org: ``` [INFO] Reactor Summary: [INFO] [INFO] Wikimedia Analytics Refinery 0.0.... [14:45:31] fdans: regarding old geowiki, let's delete old data after verifying it is present in archive in hive [14:45:34] nuria_: hola! [14:45:48] fdans: doing it with partner will be best [14:46:02] ok [14:48:33] 10Analytics, 10Operations, 10Traffic, 10Services (blocked): Add Accept header to webrequest logs - https://phabricator.wikimedia.org/T170606 (10Ottomata) Hm, either of these solutions is fine, but even if Accept isn't requested from others, it might be something fairly interesting to just include in the fu... [14:50:32] (03CR) 10Nuria: [C: 032] Add new non-wiki privacy policy [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/455567 (https://phabricator.wikimedia.org/T202792) (owner: 10Milimetric) [14:50:58] (03CR) 10Nuria: [V: 032 C: 032] Add non-wiki privacy policy [analytics/analytics.wikimedia.org] - 10https://gerrit.wikimedia.org/r/455569 (https://phabricator.wikimedia.org/T202792) (owner: 10Milimetric) [14:51:29] (03CR) 10Nuria: [V: 032 C: 032] Add non-wiki privacy policy [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/455571 (https://phabricator.wikimedia.org/T202792) (owner: 10Milimetric) [14:52:45] (03Merged) 10jenkins-bot: Update mediawiki-user-history empty-registration [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/454338 (https://phabricator.wikimedia.org/T201620) (owner: 10Joal) [14:53:51] 10Analytics: Install pyArrow in Cluster - https://phabricator.wikimedia.org/T202812 (10Ottomata) a:03Ottomata [14:56:21] 10Analytics, 10Patch-For-Review: Review Bacula home backups set for stat100[56] - https://phabricator.wikimedia.org/T201165 (10Ottomata) Ya should be fine. I'd also be fine with just declaring that stat home dirs are not backed up, and if folks specifically wanted to save stuff with more redundancy, they can... [14:57:41] ottomata: o/ [14:57:46] morningggg [14:58:20] so archiva-new *should* be ready for prime time, if you have 5 mins today can we go through the config etc.. to make sure that nothing is missing? [14:58:28] I don't have context about what was set up etc.. [15:00:40] ping ottomata [15:00:58] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Upgrade Archiva (meitnerium) to Debian Stretch - https://phabricator.wikimedia.org/T192639 (10Gehel) * Elasticsearch plugins have no direct dependency on archiva * logstash plugins only upload to archiva, @fgiunchedi has been notified, the... [15:01:46] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Upgrade Archiva (meitnerium) to Debian Stretch - https://phabricator.wikimedia.org/T192639 (10Ottomata) > For the moment I decided to split the archiva upgrade (this task) with its repository refactoring/reshaping (that will be handled in... [15:02:06] AHHH [15:05:26] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Upgrade Archiva (meitnerium) to Debian Stretch - https://phabricator.wikimedia.org/T192639 (10elukey) The main issue is that I already rsynced and configured the mirrored repo, I was under the impression that there was no clear quorum to c... [15:06:57] 10Analytics: Approximate mediawiki-history user creation dates using user-id/registration-date coherence - https://phabricator.wikimedia.org/T202899 (10JAllemandou) [15:07:39] (03CR) 10Joal: "https://phabricator.wikimedia.org/T202899" (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/454338 (https://phabricator.wikimedia.org/T201620) (owner: 10Joal) [15:15:15] 10Analytics, 10Patch-For-Review: Review Bacula home backups set for stat100[56] - https://phabricator.wikimedia.org/T201165 (10akosiaris) 05Open>03stalled [15:15:41] 10Analytics, 10Patch-For-Review: Review Bacula home backups set for stat100[56] - https://phabricator.wikimedia.org/T201165 (10akosiaris) [15:44:16] 10Analytics: Approximate mediawiki-history user creation dates using user-id/registration-date coherence - https://phabricator.wikimedia.org/T202899 (10fdans) p:05Triage>03Low [15:44:47] 10Analytics, 10Analytics-Kanban: Install pyArrow in Cluster - https://phabricator.wikimedia.org/T202812 (10fdans) [15:45:03] 10Analytics, 10Analytics-Kanban: Install pyArrow in Cluster - https://phabricator.wikimedia.org/T202812 (10fdans) p:05Triage>03Normal [15:46:20] 10Analytics, 10Page-Issue-Warnings, 10Readers-Web-Backlog (Tracking): Ingest data from PageIssues EventLogging schema into Druid - https://phabricator.wikimedia.org/T202751 (10fdans) p:05Triage>03Normal [15:52:51] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 2 others: RFC: Modern Event Platform: Schema Registry / Metadata Service - https://phabricator.wikimedia.org/T201643 (10fdans) [15:54:25] 10Analytics, 10Analytics-Kanban: Set a timeout for regex parsing in the Eventlogging processors - https://phabricator.wikimedia.org/T200760 (10fdans) [15:55:28] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Update superset (we have 0.20.6, 0.26.3 is available) - https://phabricator.wikimedia.org/T201430 (10fdans) [16:11:30] a-team: Because all good things have bad sides - https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUhkKGvjtV-dkAIsUXP-AL4ffI/preview?slide=id.g362da58057_0_1 [16:12:26] :) [16:12:37] this'll be a fun lunch-time viewing [16:12:52] milimetric: I confirm :) And the guy has some very interesting points [16:38:29] joal: that talk was great [16:38:30] hahah [16:43:49] joal: Right! [16:50:48] (03PS1) 10Nuria: Prepare for release 2.3.4 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/455594 [16:51:29] (03CR) 10Nuria: [V: 032 C: 032] Prepare for release 2.3.4 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/455594 (owner: 10Nuria) [16:54:00] (03PS1) 10Nuria: Release 2.3.4 [analytics/wikistats2] (release) - 10https://gerrit.wikimedia.org/r/455595 [16:57:50] (03CR) 10jerkins-bot: [V: 04-1] Release 2.3.4 [analytics/wikistats2] (release) - 10https://gerrit.wikimedia.org/r/455595 (owner: 10Nuria) [16:59:16] (03Abandoned) 10Nuria: Release 2.3.4 [analytics/wikistats2] (release) - 10https://gerrit.wikimedia.org/r/455595 (owner: 10Nuria) [17:29:49] 10Analytics, 10Analytics-Data-Quality, 10Contributors-Analysis, 10Product-Analytics: Resume refinement of edit events in Data Lake - https://phabricator.wikimedia.org/T202348 (10Ottomata) I tried to run a refine for Edit. There might be further complications, but the one I'm currently getting is ``` org.... [17:32:01] 10Analytics, 10Operations, 10Traffic, 10Services (blocked): Add Accept header to webrequest logs - https://phabricator.wikimedia.org/T170606 (10Ottomata) @Pchelolo we discussed this in standup today. If the data you need is small enough (can we filter on a URI?) and you only need a sample (say from a sing... [17:36:44] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10Patch-For-Review: Review analytics-in4/6 rules on cr1/cr2 eqiad - https://phabricator.wikimedia.org/T198623 (10ayounsi) ```lang=diff [edit firewall family inet filter analytics-in4 term default then] - reject; + discard; [edit firewal... [17:38:46] a-team: final ipv6 changes for the analytics vlan firewall pushed --^ [17:39:00] if you hear complains, you'll know why :) [17:40:00] 10Analytics, 10Operations, 10Traffic, 10Services (blocked): Add Accept header to webrequest logs - https://phabricator.wikimedia.org/T170606 (10Pchelolo) > If the data you need is small enough (can we filter on a URI?) and you only need a sample (say from a single cache host), AND if traffic folks don't mi... [17:41:04] * elukey off! [18:12:43] 10Analytics, 10Page-Issue-Warnings, 10Product-Analytics, 10Reading-analysis, 10Readers-Web-Backlog (Tracking): Ingest data from PageIssues EventLogging schema into Druid - https://phabricator.wikimedia.org/T202751 (10Jdlrobson) [18:55:37] joal: yt? [18:57:51] 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review: Move internal sites hosted on thorium to ganeti instance(s) - https://phabricator.wikimedia.org/T202011 (10Ottomata) a:03Ottomata [19:08:40] signing off a bit early, gonna worka couple of hours later tonight [21:15:03] 10Analytics, 10Product-Analytics, 10Reading-analysis: Assess impact of ua-parser update on core metrics - https://phabricator.wikimedia.org/T193578 (10Tbayer) >>! In T193578#4534555, @fdans wrote: > I do apologize for the delay on this. I've run the same study on bots as in https://phabricator.wikimedia.org/... [21:59:25] 10Analytics, 10Product-Analytics, 10Reading-analysis: Assess impact of ua-parser update on core metrics - https://phabricator.wikimedia.org/T193578 (10Nuria) >the last remaining step needed for answering the first question posed in the task, i.e. to replicate the impact on spider/non-spider pageviews as we a... [22:20:23] (03PS1) 10Nuria: Removing bad line left out on release branch [analytics/wikistats2] (release) - 10https://gerrit.wikimedia.org/r/455735 [22:25:03] 10Analytics, 10Product-Analytics, 10Reading-analysis: Assess impact of ua-parser update on core metrics - https://phabricator.wikimedia.org/T193578 (10Tbayer) >>! In T193578#4536526, @Nuria wrote: >>the last remaining step needed for answering the first question posed in the task, i.e. to replicate the impac...