[00:16:09] 10Analytics, 10CirrusSearch, 10Cognate, 10Discovery-Search, and 18 others: Replace TitleMoveComplet(e|ing) hooks - https://phabricator.wikimedia.org/T250023 (10DannyS712) [00:18:29] 10Analytics, 10CirrusSearch, 10Cognate, 10Discovery-Search, and 18 others: Replace TitleMoveComplet(e|ing) hooks - https://phabricator.wikimedia.org/T250023 (10DannyS712) [00:23:44] 10Analytics-Radar, 10AbuseFilter, 10Cognate, 10ConfirmEdit (CAPTCHA extension), and 28 others: Replace PageContent(Insert|Save)Complete hooks - https://phabricator.wikimedia.org/T250566 (10DannyS712) [00:25:41] 10Analytics-Radar, 10AbuseFilter, 10Cognate, 10ConfirmEdit (CAPTCHA extension), and 28 others: Replace PageContent(Insert|Save)Complete hooks - https://phabricator.wikimedia.org/T250566 (10DannyS712) [01:40:00] (03CR) 10Jenniferwang: Usage of commons files for tech tunning session metrics (031 comment) [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/606734 (https://phabricator.wikimedia.org/T247417) (owner: 10Nuria) [03:08:59] 10Analytics-Radar, 10AbuseFilter, 10Cognate, 10ConfirmEdit (CAPTCHA extension), and 27 others: Replace PageContent(Insert|Save)Complete hooks - https://phabricator.wikimedia.org/T250566 (10DannyS712) 05Open→03Resolved [03:09:09] 10Analytics, 10CirrusSearch, 10Cognate, 10Discovery-Search, and 17 others: Replace TitleMoveComplet(e|ing) hooks - https://phabricator.wikimedia.org/T250023 (10DannyS712) 05Open→03Resolved [03:09:16] 10Analytics, 10CirrusSearch, 10Cognate, 10Discovery-Search, and 17 others: Replace TitleMoveComplet(e|ing) hooks - https://phabricator.wikimedia.org/T250023 (10DannyS712) [05:13:45] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Technical contributors emerging communities metric definition, thick data - https://phabricator.wikimedia.org/T250284 (10jwang) @Nuria, @Bmueller I created a dashboard so that you can explore the pattern with other wikis. https://superset.wikimedia.org/... [06:13:36] hello folks [06:49:30] so the cause of the archiva slowness seems to be the repo groups [06:51:30] with the repo groups it is cleaner, but archiva tries every time to search an artifact in every repo listed in the group until it finds a hit [06:53:54] in archiva-old, we had a repo "mirrored" with multiple proxy connectors, that was able to quickly answer [06:55:34] we could, in theory, create multiple "mirrored" repo groups, one for each use [06:55:50] mirrored-analytics, that proxies central/spark/cloudera [06:56:00] mirrored-discovery, that proxies y/x/z [06:56:01] etc.. [06:58:02] or, even more specific, a repo called "refinery-analytics" [07:14:32] ok I created "mirrored-analytics" as test [07:14:45] it is incredibly faster sigh [07:17:25] of course it fails because of the missing "analytics-old" dependencies, that we should upload if we want to use this road [07:19:52] one alternative that I have in mind is to make nginx (that we use only to terminate TLS now on the archvia hosts) a little bit smarter, namely serving artifacts/files if already present on disk, without involving archiva at all [07:20:27] so archiva would be hit only when a dependency has not been pulled in yet [08:00:07] hi elukey - thanks for the explanation [08:00:17] quick question around mirror-groups [08:01:11] from what you wrote, archiva loops the groups and answers when it finds a hit - would the order of repos in the group help? [08:02:06] I tried, but the performances are not still good [08:02:28] you can try to use "mirrored-analytics" on stat1004, and see how fast it goes :D [08:03:01] if I manage to patch nginx to fetch artfacts/poms/etc.. on disk before contacting archiva we should resolve this problem [08:03:25] we are currently proxying all traffic to archiva, but if the files are on disk we could serve it directly [08:03:29] right I understand [08:03:55] It feels bizarre that the group is so much slower than the direct proxy [08:03:58] hm [08:04:11] http://archiva.996284.n3.nabble.com/Archiva-very-slow-td14291.html [08:04:17] this is old but support what I am saying [08:05:35] 10Analytics-EventLogging, 10Analytics-Radar, 10QuickSurveys, 10MW-1.35-notes (1.35.0-wmf.3; 2019-10-22), and 2 others: QuickSurveys EventLogging missing ~10% of interactions - https://phabricator.wikimedia.org/T220627 (10awight) >>! In T220627#6269502, @Nuria wrote: >>but in this case we're comparing two e... [08:07:21] ack elukey - this still feels weird :) [08:08:12] not sure, I wasn't really surprised, archiva seems not really a well managed apache project [08:08:16] very brittle [08:08:19] ok [08:08:21] :( [08:08:33] maybe I am wrong, but operating it is a pain [08:51:52] still trying the nginx path, the rule are more complicated than I thought [09:15:03] ok joal patch almost ready [09:15:52] ah wait it doesn't work with the repo group for sure.. [09:16:37] of course, because on disk there is no /mirrored/ path [09:16:42] since it is handled by archiva [09:17:10] it only speeds up a lot the use case of mirrored as repository [09:17:35] joal: I would be in favor of using "mirrored-analytics" and change our pom, abandoning the repo group [09:17:49] it seems the quickest way [09:18:10] but we can check together settings for the repo group, maybe there is a magic option to turn on to speed up [10:05:41] ok so I uploaded the "old" artifacts to mirrored-analytics in case we want to use it, currently testing a build on stat1004 [10:06:04] another alternative is to explicitly mention, in our pom, all the archiva repos that we need [10:06:21] but now sure how difficult it would be [10:06:27] (basically no more mirrored or repo groups needed) [10:25:27] Failed to collect dependencies at com.linkedin.camus:camus-api:jar:0.1.0-wmf9 -> org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.6.0-cdh5.4.0 [10:25:31] lovely [10:25:45] going to add this as well [10:32:33] https://archiva.wikimedia.org/#artifact~mirrored-analytics/org.apache.hadoop/hadoop-mapreduce-client-core [10:32:47] mmm it seems there, strange [10:33:24] ah it says Could not find artifact org.apache.hadoop:hadoop-project:pom:2.6.0-cdh5.4.0 [10:37:00] maybe related to the recent changes for camus, not sure [10:37:56] bizarre elukey! [10:38:01] this jar seems too old [10:38:11] maybe we forgot to update a version in camus? [10:39:11] no idea [10:39:25] elukey: I don't have strong opinion as to which option we should use (updating pom to explicitely define repos, use single mirror with plenty proxies) - I'd however like to spend some time with you later today to see if we can find a way to speed up the mirror-group [10:39:50] I can't imagine that a problem identified almost 10 years ago is still there [10:40:25] It doesn't seem to be something identified as a problem, more a "feature" [10:40:41] mwahahahb [10:40:47] ok then :) [10:41:11] but if you want to check options we can do it, I didn't find much [10:41:14] I assume listing repos is the way to go - it's also better possibly to be knowledgeable as to which repos we actually use [10:41:33] yes, and we could apply the nginx patch as well [10:41:41] yup [10:41:45] that would speed up even more builds [10:42:17] ack [10:42:23] how do you want to proceed? now/later/etc.. ? [10:42:32] (just to know when to schedule lunch :) [10:43:29] elukey: in ~1h30 ? [10:43:41] when the kids are asleep (hopefully) [10:43:46] sure :) [10:43:54] then I'll do lunch [10:43:56] ttl! [10:44:06] elukey: also - problem identified for failed unique-devices job - will send a patch [10:44:13] Thanks a lot elukey :) [10:44:15] later! [10:48:02] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Move Archiva to Debian Buster - https://phabricator.wikimedia.org/T252767 (10elukey) Summary of my understanding so far: * from reports like http://mail-archives.apache.org/mod_mbox/archiva-users/201107.mbox/%3C31F04038-99CA-4AE7-B5... [11:40:15] back! [11:52:38] 10Analytics-Radar, 10Operations, 10ops-eqiad: an-presto1004 down - https://phabricator.wikimedia.org/T253438 (10MoritzMuehlenhoff) What's the status here, any feedback from Dell on replacements etc? [12:24:21] joal: I am back as well [12:24:26] Hi elukey :) [12:24:40] elukey: can we do a quick batcave for me to see archiva admin? [12:34:02] joal: sure [12:34:18] in da cave! [12:58:41] (03PS1) 10Joal: Explicit archiva mirrored repositories in pom.xml [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/608872 (https://phabricator.wikimedia.org/T252767) [12:58:44] elukey: --^ [12:58:49] (03CR) 10jerkins-bot: [V: 04-1] Explicit archiva mirrored repositories in pom.xml [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/608872 (https://phabricator.wikimedia.org/T252767) (owner: 10Joal) [13:18:36] joal: elukey fwiw i like ^, it is nice to be able to just refer to one, but in real life with no archivia we'dl have to do that with each remote repo [13:18:52] that also allows folks not using spark to not have to consider spark deps [13:21:49] ottomata: hello :) I didn't get it you are ok with Joseph's path or not :) [13:23:00] Antoine is working on the -1 problem in https://phabricator.wikimedia.org/T252310#6271222 [13:23:31] ye! [13:23:39] it makes sense elukey sorry [13:23:40] ya [13:23:56] i was just saying that is what we would have to do IF we didn't have archiva anyway [13:24:09] it makes sense to keep the repos separate and refer to them individually [13:24:16] having a single mirror is nice [13:24:21] but not needed [13:28:00] ah ack okok :) [13:28:46] elukey: i bet you could do some nginx magic to make a /mirrored/... url check multiple places for a file [13:28:47] heheheh [13:28:55] not that you SHOULD [13:28:57] :p [13:29:13] ahahahha you will not nerd snipe me so easily :D [13:29:45] (03CR) 10Elukey: "recheck" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/608872 (https://phabricator.wikimedia.org/T252767) (owner: 10Joal) [13:29:51] (03CR) 10jerkins-bot: [V: 04-1] Explicit archiva mirrored repositories in pom.xml [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/608872 (https://phabricator.wikimedia.org/T252767) (owner: 10Joal) [13:30:09] uff [13:31:32] ah interesting, I am building on stat1004 and I see [13:31:33] Downloading: https://repository.cloudera.com/content/groups/cdh-releases-rcs/eigenbase/eigenbase-properties/1.1.4/eigenbase-properties-1.1.4.pom [13:31:51] that is a 404 link, one of the deps that we have in analytics-old [13:32:10] but mvn clean package hangs when downloading [13:32:34] or it is just failing to use the http-proxy, right [13:35:30] (03CR) 10Elukey: "recheck" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/608872 (https://phabricator.wikimedia.org/T252767) (owner: 10Joal) [13:35:37] (03CR) 10jerkins-bot: [V: 04-1] Explicit archiva mirrored repositories in pom.xml [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/608872 (https://phabricator.wikimedia.org/T252767) (owner: 10Joal) [13:36:30] (03CR) 10Hashar: "recheck T252310" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/608872 (https://phabricator.wikimedia.org/T252767) (owner: 10Joal) [13:37:25] \o/ [13:38:12] brb [13:45:06] (03CR) 10Ottomata: [C: 03+1] Explicit archiva mirrored repositories in pom.xml [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/608872 (https://phabricator.wikimedia.org/T252767) (owner: 10Joal) [13:51:42] so in analytics-old-upload we don't have the jars with the pom [13:54:10] that would explain why the mvn build is trying to pull [13:54:10] https://repository.cloudera.com/content/groups/cdh-releases-rcs/eigenbase/eigenbase-properties/1.1.4/eigenbase-properties-1.1.4.pom [13:54:16] but from cloudera [13:55:07] I can drop the repo and re-upload the jars quickly with the pom automatically generated [13:59:25] ok I dropped only the eigenbase artifact and regenerated with pom via upload, let's see [14:01:30] nope :) [14:07:42] I think that it may be refinery hive that trues [14:08:01] *tries to pull in eigenbase-properties, and for some reason the main pom.xml does not provide it [14:08:11] even if it should be in analytics-old-uploads [14:08:18] elukey: remind me why we wanted to get rid of mirrored repo in the first place? [14:10:34] ottomata: in theory to have clean repos that map to either proxies or artifacts that we care, and not a mixture of proxy dependencies (that others can't cleanly reuse directly) + uploaded files [14:10:55] I created a "mirrored-analytics" today as test [14:11:32] it will also be helpful, maybe, when/if we'll migrate to bigtop [14:11:38] is the cloudera repo still needed? [14:11:43] if not what we'll use? [14:12:38] hm aye. ya i like the idea, i just wonder if it is so much trouble to actually do if it is worth it [14:15:32] I think it is, these little issues show that we are not (in my opinion) super aware of our dependencies, and we completely rely on archiva to cache old things [14:16:18] (in the perfect scenario we could be able to build refinery without archiva) [14:29:51] hm i wonder if that is actually possible; a reason for caching the deps is to ensure that we can build our code even if a remote repo goes offline or removes a dependency [14:29:56] that we need [14:31:10] yes yes in the ideal scenario deps should be always up to date, not our case :) [14:54:31] 10Analytics, 10Product-Analytics: Streamline Superset signup and authentication - https://phabricator.wikimedia.org/T203132 (10Dzahn) We have had multiple cases of access requests now where the user requests both superset and turnilo access and after being added to an LDAP group they can access one of them but... [14:55:46] 10Analytics, 10Product-Analytics: Streamline Superset signup and authentication - https://phabricator.wikimedia.org/T203132 (10Nuria) cc @elukey to see if @Dzahn 's suggestion can be done [14:57:56] 10Analytics, 10Product-Analytics: Streamline Superset signup and authentication - https://phabricator.wikimedia.org/T203132 (10Ottomata) It would also be nice if all WMF auth could use one, but I don't think it is possible to do. IIRC, the reason we sometimes use the shell user is because the services you are... [15:03:18] Back [15:03:45] Hey, looks like our dear hashar fixed the jenkins issue \o/ [15:03:56] elukey: any news on the missing pom/jar thing? [15:04:32] nope, really weird, I tried to upload again with pom autogeneration but nothing [15:04:38] hmn [15:04:39] does it hang for you as well on stat1004? [15:04:54] trying elukey [15:05:17] for me it hangs when trying refinery hive [15:05:29] downloading from repository.cloudera.com [15:06:46] same for me elukey [15:07:17] must be a config issue as the download is not from archiva-mirrored-cloudera [15:07:28] elukey: https_proxy set? [15:07:49] cdanis: we're trying to download from archiva instead of external repos [15:08:01] ah ok [15:08:18] (03CR) 10Nuria: [C: 03+2] Usage of commons files for tech tunning session metrics (031 comment) [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/606734 (https://phabricator.wikimedia.org/T247417) (owner: 10Nuria) [15:10:40] cdanis: yes for some reason it uses the cloudera repo, a nice maven dependency gift :D [15:11:14] elukey: the jar seems available in maven :S [15:11:17] https://mvnrepository.com/artifact/eigenbase/eigenbase-properties/1.1.4 [15:11:30] Why would our stuff look for it in cloudera? [15:11:51] ah no - sorry - missed "Note: this artifact it located at Spring Plugins repository (https://repo.spring.io/plugins-release/)" [15:12:04] yep [15:12:55] hm - And how come we try downloading from cloudera repo???? [15:13:57] joal: I suspect it is buried in some pom.xml of the refinery-hive deps [15:14:11] maybe it was in cloudera, and now it is not [15:14:13] elukey: from archiva config, the jar should be accessible [15:14:24] in analytics-old [15:14:31] will try to make teh dep explicit [15:14:39] it seems as if it doesn't touch analytics-old [15:14:46] but only the mirror- ones [15:14:52] yeah [15:15:12] maybe it pulls the analytics-old index not finding what it needs [15:16:47] heh, i have some refinery-source issues due to failed deploys yesterday [15:17:02] apparently if the current SNAPSHOTs are not in archiva...i can't build??? [15:17:03] I can imagine [15:17:05] i dunno. [15:17:14] ottomata: have you pulled the last master? [15:17:22] yes, it has 0.0.129 stuff [15:17:24] should I use that? [15:17:29] ottomata: there have been cleanup [15:17:36] ah [15:17:38] reverts [15:17:38] ok [15:17:42] should build 0.0.129-SNAPSHOT [15:17:59] also, if you have local tags v0.0.129 ou can drop them :) [15:18:23] (03PS2) 10Ottomata: Remove unused custom avro camus classes [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/608725 [15:18:44] elukey: I don't understand how we endup trying to download from cloudera repo - Do we have anything settup for thaT? [15:18:57] got it thank you [15:18:58] elukey: I thought everything was from mirror:S [15:19:02] (03PS16) 10Ottomata: Add classes to use EventStreamConfig with EventSchemaLoader to aide in event ingestion tasks [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603582 (https://phabricator.wikimedia.org/T251609) [15:19:36] joal: not in the poms that we define afaics [15:23:01] (03CR) 10jerkins-bot: [V: 04-1] Add classes to use EventStreamConfig with EventSchemaLoader to aide in event ingestion tasks [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603582 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [15:28:39] joal: what happens if repos are listed in a dependency's pom? Is it possible? [15:28:48] if so, it might explain the cloudera repo [15:29:01] possible - but it feals so weird!P [15:29:21] (03CR) 10Nuria: [V: 03+2 C: 03+2] Usage of commons files for tech tunning session metrics [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/606734 (https://phabricator.wikimedia.org/T247417) (owner: 10Nuria) [15:29:44] joal: the other thing is - are you able to build locally or same issue? [15:30:01] (03CR) 10Jenniferwang: Usage of commons files for tech tunning session metrics (031 comment) [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/606734 (https://phabricator.wikimedia.org/T247417) (owner: 10Nuria) [15:30:01] I am trying but I have some other issues, scala related, probably my laptop is not configured correctly [15:30:04] elukey: I think locally it'll be fine as there is no proxy blocking [15:30:48] no I mean if locally it still does the weird repo.cloudera pull, maybe for some weird reason the /etc/maven/settings.xml are interfering [15:31:20] on stat1004 I basically applied manually https://gerrit.wikimedia.org/r/c/operations/puppet/+/608879 [15:31:57] maybe I can try to mess with it [15:32:02] elukey: i don't have any items for ops sync, but am happy to do it if you want to [15:32:11] i am guessing your head might be deep in archiva atm [15:32:22] ottomata: I have an unrelated thing to show you if you have time, 1 min :) [15:32:56] otherwise another time [15:33:01] (not work related) [15:33:32] elukey: I built locally, without having to download eigenbase (while I had dropped it from local repo) [15:33:35] * joal is getting mad [15:33:56] elukey: ok [15:33:59] bc! [15:34:43] elukey: all my downloads were made from mirrored stuff [15:36:35] elukey: \o/! I got it - mavel folder was not cleanly removed in my previous run (needed -f), new run after deletion tries to download from mirror-central, then mirror-spark, then mirror-cloudera, and finally analytics-old, where it finds the jar [15:36:50] So there must be something wrong with our settings [15:37:39] on stat 1004 [15:39:29] niceeeee [15:39:32] 10Analytics-Kanban, 10Analytics-Radar, 10Product-Analytics, 10Patch-For-Review: Definition of not text content metrics for tuning session (rich media, images, etc) - https://phabricator.wikimedia.org/T247417 (10Nuria) data will be available at: https://analytics.wikimedia.org/published/datasets/periodic/re... [15:39:45] good we are getting to the problem then [15:39:59] Now why the heck is stat1004 willing to download from cloudera directly???? [15:40:34] joal: maybe I can try to remove settings.xml from etc and see [15:40:43] works for me elukey [15:40:51] (03CR) 10Ottomata: Add classes to use EventStreamConfig with EventSchemaLoader to aide in event ingestion tasks (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603582 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [15:41:23] (03PS17) 10Ottomata: Add classes to use EventStreamConfig with EventSchemaLoader to aide in event ingestion tasks [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603582 (https://phabricator.wikimedia.org/T251609) [15:42:07] (03PS3) 10Ottomata: Remove unused custom avro camus classes [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/608725 [15:42:58] (03PS18) 10Ottomata: Add classes to use EventStreamConfig with EventSchemaLoader to aide in event ingestion tasks [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603582 (https://phabricator.wikimedia.org/T251609) [15:43:38] (03CR) 10Ottomata: Overloaded methods to make working with default Refine related classes easier (033 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/607788 (owner: 10Ottomata) [15:47:09] elukey: still no chance :( [15:51:22] joal: it worked for me :O :O [15:51:25] build succeded [15:51:26] elukey: I think I nailed it - /o\ [15:51:28] on stat1004 [15:51:31] hehehe :) [15:51:33] same time [15:51:35] nice [15:52:33] (03CR) 10jerkins-bot: [V: 04-1] Add classes to use EventStreamConfig with EventSchemaLoader to aide in event ingestion tasks [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603582 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [15:52:55] here is what I found: when a download has been done before (or fialed), maven remembers the location of the downloaded stuff - since we have changed locations, we needed to drop .m2/repository [15:53:17] elukey: https://stackoverflow.com/questions/5271707/maven-tries-to-download-dependency-despite-it-existing-in-local-repository/5274224 [15:53:24] but from the local home dir right? [15:53:38] because I did it every time [15:53:43] (03CR) 10jerkins-bot: [V: 04-1] Remove unused custom avro camus classes [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/608725 (owner: 10Ottomata) [15:55:09] correct elukey - local-home repo [15:55:16] so the /etc/maven/settings.xml come from https://gerrit.wikimedia.org/r/c/operations/puppet/+/170668 [15:55:22] that predates me :D [15:55:29] ah interesting - I also have a test failing [15:55:44] (03PS4) 10Ottomata: Remove unused custom avro camus classes [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/608725 [15:55:46] (03PS19) 10Ottomata: Add classes to use EventStreamConfig with EventSchemaLoader to aide in event ingestion tasks [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603582 (https://phabricator.wikimedia.org/T251609) [15:56:30] ottomata: do you think that /etc/maven/settings.xml is still needed? [15:56:43] for some reason it is what it made my build failing [15:56:50] on stat1004 [15:57:12] elukey: we can try with the original patch you had (removing mirror only0 [15:58:02] joal: I applied it earlier on stat1004 (with puppet disabled) but didn't count much (releases was still mentioned as mirror) [15:58:55] I would go as far as completely remove the file [15:59:37] we have it deployed on 72 hosts [15:59:40] an-airflow1001.eqiad.wmnet,an-coord1001.eqiad.wmnet,an-launcher1002.eqiad.wmnet,an-tool1006.eqiad.wmnet,an-worker[1078-1095].eqiad.wmnet,analytics[1030-1031,1033-1038,1040,1042-1077].eqiad.wmnet,stat[1004-1008].eqiad.wmnet [15:59:52] but basically it is helpful only on stat100x [16:00:12] (I am not seeing any use case to run mvn on hadoop worker nodes :D) [16:02:39] ping ottomata standdduppppp [16:03:19] AHHHH [16:03:36] elukey: [16:03:37] Downloaded: https://archiva.wikimedia.org/repository/mirror-maven-central/commons-codec/commons-codec/maven-metadata.xml (712 B at 9.0 KB/sec) [16:03:40] Downloading: https://oss.sonatype.org/content/repositories/snapshots/commons-codec/commons-codec/maven-metadata.xml [16:03:44] :( [16:03:51] for refinery-jobs [16:04:10] very interesting, snapshots? [16:04:20] ? [16:04:24] Ah! [16:04:30] indeed, didn't notice [16:04:40] I am re-running the build as well [16:05:02] I see this joal [16:05:02] Downloading: https://oss.sonatype.org/content/repositories/snapshots/commons-codec/commons-codec/maven-metadata.xml [16:05:05] Downloaded: https://archiva.wikimedia.org/repository/releases/commons-codec/commons-codec/maven-metadata.xml (282 B at 6.7 KB/sec) [16:05:11] and now it is hanging, but before it continued [16:05:12] mmmm [16:05:30] elukey: I assume the ordering of repos has a play here [16:05:51] it is hanging though, let's see if it unblocks [16:06:00] elukey: no chances IMO [16:06:04] :) [16:06:12] you don't trust me I know [16:06:18] :-P [16:06:20] it is a trend recently [16:06:25] :D [16:06:26] mwahahaha :) [16:09:26] joal: the conn times out and the build proceeds [16:09:35] ok :) [16:09:38] but whyyyy [16:09:45] folder order I guess [16:09:57] no I mean why it doesn't fail [16:10:13] maybe the file was present, or some versiohn of it? [16:11:20] not sure - we could setup a connection-timeout smaller? [16:14:35] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: permanent links in wikistats don't (always) work - https://phabricator.wikimedia.org/T254076 (10Milimetric) Sorry, this was deployed a while back and we forgot to move it to Done. My bad, and thanks again for the great bug report @Effeietsanders [16:23:02] 10Analytics: EventGate thottling and DOS prevention - https://phabricator.wikimedia.org/T256891 (10Nuria) [16:28:09] ok joal something interesting [16:28:42] in theory https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release-docker/45/consoleFull should be a recent build done with archiva-old [16:28:46] (correct me if I am wrong) [16:28:49] I downloaded the file [16:28:54] and grepped [16:29:01] 18:58:06 [INFO] [INFO] Downloading from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/commons-codec/commons-codec/maven-metadata.xml [16:29:06] 18:58:07 [INFO] [INFO] Downloading from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/commons-codec/commons-codec/1.15-SNAPSHOT/maven-metadata.xml [16:29:19] wow [16:29:28] and timeout? [16:30:10] no trace afaics [16:30:24] but I don't see any "Downloaded" entry as well [16:30:39] hm [16:49:20] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: permanent links in wikistats don't (always) work - https://phabricator.wikimedia.org/T254076 (10Nuria) 05Open→03Stalled [16:50:42] (03CR) 10Joal: [C: 03+2] "Merging to test deploy with jenkins" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/608872 (https://phabricator.wikimedia.org/T252767) (owner: 10Joal) [16:51:18] !log remove /etc/maven/settings.xml from all analytics nodes that have it [16:51:20] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:56:29] (03Merged) 10jenkins-bot: Explicit archiva mirrored repositories in pom.xml [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/608872 (https://phabricator.wikimedia.org/T252767) (owner: 10Joal) [16:56:29] * elukey out for a run, be back later [16:58:06] \o/ jenkins is back with us [16:58:21] ok trying to release refinery-source to archiva, version 3 [16:58:32] !log trying to release refinery-source 0.0.129 to archiva, version 3 [16:58:33] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:00:03] Starting build #52 for job analytics-refinery-maven-release-docker [17:11:10] Yippee, build fixed! [17:11:11] Project analytics-refinery-maven-release-docker build #52: 09FIXED in 11 min: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release-docker/52/ [17:31:25] 10Analytics: Refine event pipeline at this time refines data in hourly partitions without knowing if the partition is complete - https://phabricator.wikimedia.org/T252585 (10Ottomata) [17:57:49] hm elukey how is pcc supposed to work with keytab files? [17:57:50] https://puppet-compiler.wmflabs.org/compiler1003/23615/an-launcher1002.eqiad.wmnet/change.an-launcher1002.eqiad.wmnet.err [17:58:01] do we need a dummy file for an-launcher1002 somewhere? [18:13:13] 10Analytics, 10Analytics-Kanban, 10Event-Platform: Backfill wdqs_external_sparql_query without filtering on meta.domain - https://phabricator.wikimedia.org/T256797 (10Ottomata) ` 20/07/01 17:55:36 INFO Refine: Successfully refined 1535 of 1535 dataset partitions into table `event`.`wdqs_external_sparql_query... [18:16:03] ok - following up the dpeloy [18:16:47] Starting build #19 for job analytics-refinery-update-jars-docker [18:17:03] (03PS1) 10Maven-release-user: Add refinery-source jars for v0.0.129 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/608921 [18:17:03] Project analytics-refinery-update-jars-docker build #19: 09SUCCESS in 17 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars-docker/19/ [18:17:32] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging for deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/608921 (owner: 10Maven-release-user) [18:20:37] !log Deploy refinery using scap [18:20:38] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:22:59] quick note on unique-devices monthly job: computation time is divided by roughly 3 [18:23:09] not bad [18:25:11] (03PS1) 10Joal: Fix unique-devices per project-family monthly job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/608922 [18:26:24] (03CR) 10Nuria: [C: 03+1] "I should have caught that" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/608922 (owner: 10Joal) [18:27:01] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging hotfix for deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/608922 (owner: 10Joal) [18:28:13] !log Deploy refinery using scap after hotfix [18:28:14] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:28:44] nuria: note on unique-devices data change - I have not documented the analysis - will do this end-of-week [18:29:58] 10Analytics, 10Analytics-Kanban, 10Event-Platform: Backfill wdqs_external_sparql_query without filtering on meta.domain - https://phabricator.wikimedia.org/T256797 (10JAllemandou) `refinery-source-0.0.129.jar` is now available :) [18:31:30] Ah crap - I have an issue deploying ottomata [18:31:41] ottomata: Could not resolve hostname an-launcher1001.eqiad.wmnet [18:31:41] oh ? [18:31:45] oh! [18:31:52] we need to move to an-launcher1002 [18:31:52] must be wrong in scap targets [18:32:28] ottomata: currently deploying everywhere else, then we can manually do an-launcher1002 [18:34:30] (03PS1) 10Ottomata: Remove an-launcher1001 from scap targets [analytics/refinery/scap] - 10https://gerrit.wikimedia.org/r/608925 [18:34:43] joal: ^ [18:35:26] (03CR) 10Joal: [V: 03+2 C: 03+2] "LGTM! Merging for next deploy" [analytics/refinery/scap] - 10https://gerrit.wikimedia.org/r/608925 (owner: 10Ottomata) [18:35:38] merged ottomata [18:35:44] thanks a lot [18:36:41] ottomata: simmilar error for thin env: notebook1003.eqiad.wmnet [18:37:08] I think we should remove notebook1004 at the same time as it's not needed anymore [18:37:50] aye joal amended [18:38:00] ottomata: it was already merged :( [18:38:02] OH you merged [18:38:07] sorry for that [18:39:08] (03PS1) 10Ottomata: Remove decommissioned notebook hosts from scap targets [analytics/refinery/scap] - 10https://gerrit.wikimedia.org/r/608926 [18:40:51] joal: ^ [18:40:53] ottomata - I merge this one too :) [18:41:03] (03CR) 10Joal: [V: 03+2 C: 03+2] "LGTM - Merging as well" [analytics/refinery/scap] - 10https://gerrit.wikimedia.org/r/608926 (owner: 10Ottomata) [18:41:21] !log deploy refinery to HDFS [18:41:22] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:42:17] ottomata: yes correct we need the dummy files in the labs_private [18:42:33] joal: how did you get the error for notebook1003/4? running the lightweight env? [18:42:38] I forgot about that sorry [18:43:02] anyway, builds worked \o/ [18:43:08] all good so far right? [18:43:16] (except the scap targets sorry) [18:48:23] ah also sqoop is running on launcher1002 [18:48:40] resource usage look very good [18:48:41] https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&refresh=5m&var-server=an-launcher1002&var-datasource=thanos&var-cluster=analytics&from=now-24h&to=now [18:48:54] (03CR) 10Nettrom: "Reply to question about removing the hashing." (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/607615 (https://phabricator.wikimedia.org/T255501) (owner: 10Nettrom) [18:49:32] elukey: I got error `connection to notebook1003.eqiad.wmnet failed and future stages will not be attempted for this target` [18:49:57] elukey: I noticed sqoop was going faster :) I'm glad :) [18:50:01] thanks again elukey :) [18:50:19] gooood [18:50:20] :) [18:50:37] elukey: and indeed as ou noticed, released succeeded :) [18:50:43] a good end of day :) [18:52:03] Ok, restarting jobs [18:52:24] !log Kill/Restart unique_devices-per_project_family-monthly-coord after fix [18:52:25] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:55:09] !log kill/restart mediawiki-history-denormalize-coord after skewed-join strategy update [18:55:10] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:56:30] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Technical contributors emerging communities metric definition, thick data - https://phabricator.wikimedia.org/T250284 (10Bmueller) From the sync with @jwang today: * To gain insights how bots contributed to the content while a project grew, it would be g... [18:57:03] !log kill/restart mediawiki-wikitext-history-coord and mediawiki-wikitext-current-coord for bz2 codec update [18:57:05] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:59:59] !log kill/restart pageview-druid jobs (hourly, daily, monthly) for in_content_namespace field update [19:00:00] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:04:33] !log Kill/restart webrequest-load-bundle for mobile-pageview update [19:04:34] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:06:26] Ok all restarts done, no error so far - gone for diner, back to double check in a bit [19:21:41] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Technical contributors emerging communities metric definition, thick data - https://phabricator.wikimedia.org/T250284 (10jwang) > To gain insights how bots contributed to the content while a project grew, it would be great to have data with longer history... [19:58:31] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Add examples to all event schemas - https://phabricator.wikimedia.org/T242454 (10Ottomata) Phew, ok! Should be done with schemas/event/primary! Now on to schemas/event/secondary repo. [20:25:20] bearloga: we're back in track on mobile pageviews :) [20:25:34] Ok all good so far - dropping for tonight :) [20:26:00] joal: woot! woot! thanks!!! [20:26:33] :)