[04:59:31] good morning and RIP our inboxes :) [06:57:03] 10Analytics, 10Analytics-Cluster, 10Operations, 10ops-eqiad: Renamed notebook1003 to an-launcher1002 - https://phabricator.wikimedia.org/T256397 (10elukey) Updated wikitech in https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Rename_while_reimaging with the procedure that I followed (mentioning also to... [06:57:26] hello hello [07:04:24] very nice metrics from an-launcher1002 [07:04:25] https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&refresh=5m&var-server=an-launcher1002&var-datasource=eqiad%20prometheus%2Fops&var-cluster=analytics&from=now-24h&to=now [07:04:57] the host is now using a lot of cached memory, and it is leveraging more cores [07:05:03] I think it is a win :) [07:16:05] 10Analytics, 10Analytics-Cluster: Repurpose notebook100[3,4] - https://phabricator.wikimedia.org/T256363 (10elukey) The next step is to rename notebook1004 to something like an-airflow/an-scheduler. [07:28:21] Good morning team [07:30:51] bonjour [07:31:04] hellooo [07:36:22] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban: Repurpose notebook100[3,4] - https://phabricator.wikimedia.org/T256363 (10elukey) [09:10:18] joal: just seen https://www.nytimes.com/interactive/2020/us/coronavirus-spread.html, wow [09:10:31] I am really impressed with that data viz [09:10:35] thanks for sharing [09:10:38] elukey: impressive indeed :) [09:12:14] my laptop fan started to blow like an airplane but maybe it was coincidence :D [09:14:42] elukey: I'm sure it is not :) [09:15:01] elukey: javascript for fresh air! [09:26:39] joal: one qs if you have time [09:26:43] sure [09:27:04] I changed refinery source's pom.xml with archiva-new in all places, removed ~/.m2 and executed mvn clean package [09:27:08] but I keep seeing [09:27:08] Downloading: https://archiva.wikimedia.org/repository/mirrored/org/scala-tools/maven-scala-plugin/2.15.2/maven-scala-plugin-2.15.2.pom [09:27:16] hm [09:28:05] did you see archiva-new when you executed your build? [09:28:55] checking again elukey [09:29:17] elukey: my laptop went crazy about halfway through that page as well, I wonder if that should be sent to the covid list [09:29:52] RhinosF1: feel free to do it, I am not on it but it could be an interesting thing to post! [09:30:00] elukey: will do [09:30:06] elukey: it uses archiva not new for download - MEH [09:31:03] weird [09:31:18] sent [09:31:23] ack! [09:31:31] elukey: could it be from archiva-new config? [09:31:50] joal: what do you mean? [09:31:56] https://lists.wikimedia.org/pipermail/covid-19-stats/2020-June/000026.html [09:31:59] (trying to understand what could be wrong) [09:32:07] archiva-new saying: you should download from archiva [09:32:24] ahhhh [09:33:01] in theory no, but I'd expect maven to write archiva-new anyway when printing "downloading etc.." [09:33:08] indeed [09:34:59] elukey: could it be a reverse-proxy thing? [09:35:02] I can't imagine how [09:36:02] I see me and you in the archiva request log on 1001 [09:36:16] this is very bizarre! [09:36:16] (the backend for archiva.wikimedia.org) [09:37:31] elukey: I got I think - /usr/share/maven/conf/settings.xml [09:38:51] ahhhhhhh [09:38:59] totally new to me [09:39:58] elukey: I copied the file to ~/.m2/settings.xml, updated the value to add -new - working ! [09:40:18] Now downloading from new :) [09:41:40] super [09:41:44] really nice discovery [09:51:34] Failed to execute goal on project refinery-spark: Could not resolve dependencies for project org.wikimedia.analytics.refinery.spark:refinery-spark:jar:0.0.129-SNAPSHOT: Could not find artifact graphframes:graphframes:jar:0.7.0-spark2.3-s_2.11 in system-wide-wmf-mirrored-default (https://archiva-new.wikimedia.org/repository/mirrored/) [09:51:40] so I think I have to add another repo [09:51:48] and add it to mirrored [09:52:03] hm [09:52:41] elukey: we should bump grapframes to 2.4 :) [09:53:42] getting Lino - back after lunch [09:54:29] added the spark repo, there was a proxy configured that was not mapped [09:54:34] trying to rebuild [10:18:28] Could not resolve dependencies for project org.wikimedia.analytics.refinery.hive:refinery-hive:jar:0.0.129-SNAPSHOT: The following artifacts could not be resolved: org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde [10:20:47] 10Analytics, 10Analytics-EventLogging, 10QuickSurveys, 10WMDE-Technical-Wishes-Team: What proportion of our visitors have blocked EventLogging? - https://phabricator.wikimedia.org/T256464 (10awight) [10:32:07] ^ I'd be curious to know if there's an anecdotal answer to this already? [10:35:01] not sure that we can make comparison easily, maybe using virtual pageviews and pageviews? [10:35:09] total speculation, I am very ignorant about it [10:35:19] you have to wait the test of the team :D [10:35:56] 10Analytics, 10Analytics-EventLogging, 10QuickSurveys, 10WMDE-Technical-Wishes-Team: What proportion of our visitors have blocked EventLogging? - https://phabricator.wikimedia.org/T256464 (10kostajh) @nettrom_WMF may remember the answer to this (for logged-in users anyway), because we looked at this not th... [10:38:26] * elukey lunch! [10:53:53] 10Analytics, 10Analytics-EventLogging, 10QuickSurveys, 10WMDE-Technical-Wishes-Team: QuickSurveys should show an error when response is blocked - https://phabricator.wikimedia.org/T256463 (10awight) It doesn't look like we can detect the failed request. EventLogging is hardcoded to use asynchronous `sendB... [11:01:14] Yeah that sort of approach seems like a good idea for detecting all kinds of problems. We did something similar recently and discovered an astonishing number of users without JS at all, fwiw. [11:02:02] 38% of anons or "new users" reaching the edit conflict workflow seem to have no JS running in their browser! https://github.com/wikimedia/analytics-wmde-TW-edit-conflicts/blob/master/reports/Small%20default%20questions.ipynb [11:20:34] 10Analytics-EventLogging, 10Analytics-Radar, 10QuickSurveys, 10MW-1.35-notes (1.35.0-wmf.3; 2019-10-22), and 2 others: QuickSurveys EventLogging missing ~10% of interactions - https://phabricator.wikimedia.org/T220627 (10awight) Late entry here--reading through the comments, I think the only hypothesis whi... [11:25:16] 10Analytics, 10Analytics-EventLogging, 10QuickSurveys, 10WMDE-Technical-Wishes-Team: What proportion of our visitors have blocked EventLogging? - https://phabricator.wikimedia.org/T256464 (10awight) 05Open→03Resolved a:03awight This was answered precisely by @AndyRussG in T236834#5696044: When compar... [11:25:18] 10Analytics, 10Analytics-EventLogging, 10QuickSurveys, 10WMDE-Technical-Wishes-Team: QuickSurveys should show an error when response is blocked - https://phabricator.wikimedia.org/T256463 (10awight) [11:28:33] Will EventLogging 3.0 include a synchronous client call? I have a use case, QuickSurveys, where it would be useful to know whether the log was received or not. [11:30:15] * awight notes ottomata is out [11:31:35] elukey: I tried to repro the build issue for refinery-spark, and it didn't fail for me :( [12:31:25] joal: o/ [12:31:38] so for refinery spark it should be fixed, I added a repo [12:31:53] (there was a proxy that I didn't add to the mirrored repo group) [12:32:05] but now it fails, for me, with refinery-hive [12:32:18] see above about pentaho [12:32:23] does it happen the same for your? [12:39:58] I am wondering if this is an example of artifact that we manually uploaded to mirrored [12:40:08] from a quick search it seems contained in http://conjars.org/repo/ [12:41:06] no this seems to be another generic repo sorry [12:57:16] but one would expect to find org.pentaho etc.. in the cloudera repo [12:57:18] in theory [13:00:08] exactly, one WOULD expect https://repository.cloudera.com/artifactory/cloudera-repos/org/ [13:00:43] hi teammmm [13:01:37] hola marcel! [13:04:41] I found https://repo.spring.io/plugins-release/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/ [13:04:53] maybe we could mirror https:/repo.spring.io [13:16:01] mforns: when you have a sec, let's check if RU jobs are good on an-launcher1002 [13:16:07] if so I'll drop the old VM [13:16:15] elukey: ok [13:16:25] elukey: doing now [13:17:22] <3 [13:24:34] elukey: I see pingback reports executing, some of them have finished successfully, but others are still failing, I believe the problem is data size [13:25:03] but overall we are good with the move right? [13:25:35] elukey: many right, some reports have not run yet, because they are weekly or monthly [13:25:57] ahh okok so I'll wait for decomming [13:26:26] no no, I can check the logs for them, one sec [13:43:21] reportupdater-ee seems to be stuck since 2018-10 so, I guess we can deactivate that job.. [13:43:38] lol [13:44:17] I think it's an old one, don't know why it was runnin... [13:45:10] the reportupdater-ee-beta-features seems to behave the same as before the migration (erroring because of missing user_properties table in labsdbs) [13:45:14] so all "good" [13:48:30] :) [13:49:39] elukey: all other jobs seem fine! [13:51:08] gooood! [14:08:37] mmm I tried to add the repo.spring.io repo as test, but https://archiva-new.wikimedia.org/repository/mirrored/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom leads to 404 [14:09:31] but I see https://repo.spring.io/plugins-release/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/ [14:09:47] ah maybe that plugins-release [14:10:06] sigh [14:13:28] * elukey removes the spring repo [14:21:46] so https://archiva.wikimedia.org/#artifact/org.pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde seems not coming neither from central nor from cloudera [14:27:50] the other alternative could be to create a repo to which we can upload artifacts, and copy old deps like --^ from archiva1001 [14:27:58] then we put it under the mirrored repo group [14:28:19] basically cherry picking only the deps that we don't get from central or cloudera [14:28:48] lemme see if I can hack something [14:35:57] works! [14:36:02] I see now another dep missing [14:52:02] ok now I get to something more interesting [14:52:10] [ERROR] /srv/home/elukey/source/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/refine/RefineFailuresChecker.scala:16: error: RefineFailuresChecker is already defined as object RefineFailuresChecker [14:54:21] [INFO] object RefineFailuresChecker extends LogHelper with ConfigHelper { [14:54:23] [INFO] ^ [14:54:25] now I am really confused [14:54:27] [ERROR] one error found [15:11:11] but the downloading part looks good now [15:35:25] Hi elukey - excuse I was gone for kids [15:35:54] no need for excuses :) [15:36:27] elukey: something I know we have done with Andrew before is adding mirrors temporary, for needed deps to be cached in local archiva, then mirror got removed [15:41:58] joal: so I created a repo called "analytics-old" containing a few deps not in public repos but on archiva1001, and it seems working, all downloads are fine [15:42:03] but then I get the above error [15:42:39] elukey: currently reying to repro [15:48:07] elukey: can't repro - build has worked for me [15:48:18] elukey: could it be you're not using the last master? [15:50:12] weird, i just checked out the source repo [15:50:21] :( [15:51:02] and you used mvn clean package right? [15:51:23] elukey: correct, with the -s for my setttings.xml file [15:54:43] ah snap my repo had a file not tracked, PEBCAK [15:54:51] ok ok mistery solved, stupid luca [15:55:05] joal: ok so archiva-new seems to work :) [15:55:09] \o/ [15:55:20] is it ok for you if we keep the 'analytics-old' repo? [15:55:57] in theory it will be super easy in the future to drop it when not needed anymore [15:56:26] elukey: it contains only the jars missing for current compile, right? [15:56:52] correct [15:57:14] there are 3, I think one is not needed but it can stay there [15:57:32] elukey: works for me - maybe we could make a more explicit name? analytics-manually-uploaded ? [15:57:39] even if not, works for me :) [15:58:01] I'd need to recreate the repo etc.., I'd avoid it if possible :) [15:58:08] ack elukey :) [15:58:25] will we need the cloudera repo when after bigtop? [15:58:35] elukey: I guess not ? [15:59:15] it is something to check eventually, not sure where it is needed currently [15:59:34] I mean, what specific deps are needed and for what [15:59:42] ack [16:01:44] all right, we can probably switch on monday then [16:01:51] thanks a lot for checking :) [16:02:10] np elukey - thanks for doing the work!!!! [16:04:44] also - backfilling job for pageview_actor_hourly done - data since beginning of June is available [16:05:00] elukey: I'm planning on providing a patch for deletion after 90 days early next week [16:05:15] okok [16:44:50] going afk, have a good weekend folks! [16:44:51] o/ [16:44:58] mforns: good luck! :) [16:44:59] bye elukey :) [16:50:34] 10Analytics, 10Analytics-EventLogging, 10QuickSurveys, 10WMDE-Technical-Wishes-Team: What proportion of our visitors have blocked EventLogging? - https://phabricator.wikimedia.org/T256464 (10nettrom_WMF) Thanks for the ping @kostajh! I'm going to throw a bit of nuance on the number that @AndyRussG reports... [18:38:44] 10Analytics, 10Analytics-Cluster, 10Operations, 10ops-eqiad: Renamed notebook1003 to an-launcher1002 - https://phabricator.wikimedia.org/T256397 (10wiki_willy) a:03Jclark-ctr [19:38:12] :] [19:50:11] Gone as well - safe trip mforns - see you in a couple of days :) [20:42:19] bearloga: yello! yt? [20:42:21] looking at https://wikitech.wikimedia.org/w/index.php?title=Event_Platform%2FInstrumentation_How_To&type=revision&diff=1870649&oldid=1869417 [20:42:31] ottomata: howdy! [20:43:57] i had thought i had set up mw vagrant and eventlogging role there to always produce an event even if its stream was not configured [20:44:59] https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/EventLogging/+/master/modules/ext.eventLogging/core.js#325 [20:44:59] and [20:45:32] https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/vagrant/+/master/puppet/modules/role/templates/eventlogging/EventLogging.php.erb#12 [20:47:13] bearloga: ^ [20:51:29] ottomata: hm… does the debug mode work if using mw vagrant anonymously? also in that case I need to update the sampling config patch to have the same behavior because I have it mark streams-not-in-config as out-of-sample https://gerrit.wikimedia.org/r/c/mediawiki/extensions/EventLogging/+/602739/9/modules/ext.eventLogging/core.js#434 [20:55:38] bearloga: i guess we can make it do whatever you think is best! [20:55:52] i was trying to make EventLogging work in dev without EventStreamConfig [20:55:54] in mw-vagrant [20:56:14] evengate there is also set up to just produce whatever it gets. EventGate will still validate the schema, but it will allow any event to any stream [21:01:26] bye alll! [21:01:36] see you in 2 weeks :] [21:15:25] 10Analytics, 10Analytics-Kanban, 10Cloud-VPS, 10Puppet: Puppet failing on wikistats.analytics.eqiad.wmflabs: /usr/local/sbin/x509-bundle error - https://phabricator.wikimedia.org/T255464 (10bd808) Still busted: `lines=10,lang=shell-session root@wikistats:~# puppet agent -tv Info: Using configured environme... [21:50:41] 10Analytics, 10Product-Analytics: API pageview counts for 'Mobile app' are incorrect since switch to mobile-html - https://phabricator.wikimedia.org/T256508 (10kzimmerman) p:05Triage→03High [22:02:57] 10Analytics-Kanban, 10Better Use Of Data, 10Product-Analytics: Superset Updates - https://phabricator.wikimedia.org/T211706 (10SNowick_WMF) Low priority: This looks to be a bug in Superset (https://github.com/apache/incubator-superset/issues/9229) but want to report I am seeing the same issue. When using Tim... [22:11:02] 10Analytics-Kanban, 10Better Use Of Data, 10Product-Analytics: Superset Updates - https://phabricator.wikimedia.org/T211706 (10SNowick_WMF) 05Open→03Resolved a:03SNowick_WMF [22:41:21] 10Analytics, 10Product-Analytics (Kanban): PageviewDefinition should detect /api/rest_v1/page/mobile-html requests as pageviews - https://phabricator.wikimedia.org/T256514 (10mpopov) [22:42:46] (03PS1) 10Bearloga: Label mobile-html endpoint requests as app pageviews [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/608100 (https://phabricator.wikimedia.org/T256514) [22:42:55] (03CR) 10jerkins-bot: [V: 04-1] Label mobile-html endpoint requests as app pageviews [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/608100 (https://phabricator.wikimedia.org/T256514) (owner: 10Bearloga) [22:44:04] (03CR) 10Bearloga: "recheck" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/608100 (https://phabricator.wikimedia.org/T256514) (owner: 10Bearloga) [22:46:41] 10Analytics, 10Product-Analytics: New app pageview definition needs to be deployed - https://phabricator.wikimedia.org/T256515 (10mpopov) [22:51:10] 10Analytics, 10Product-Analytics: Re-process webrequests from 2020-05-18 so that page views from latest Wikipedia app releases are counted - https://phabricator.wikimedia.org/T256516 (10mpopov) [22:51:43] 10Analytics, 10Product-Analytics: New app pageview definition needs to be deployed - https://phabricator.wikimedia.org/T256515 (10mpopov) [22:51:46] 10Analytics, 10Patch-For-Review, 10Product-Analytics (Kanban): PageviewDefinition should detect /api/rest_v1/page/mobile-html requests as pageviews - https://phabricator.wikimedia.org/T256514 (10mpopov) [22:51:58] 10Analytics, 10Product-Analytics: Re-process webrequests from 2020-05-18 so that page views from latest Wikipedia app releases are counted - https://phabricator.wikimedia.org/T256516 (10mpopov) [22:52:00] 10Analytics, 10Product-Analytics: New app pageview definition needs to be deployed - https://phabricator.wikimedia.org/T256515 (10mpopov)