[01:57:10] 10Analytics: Presto Error: parquet file declares column as wrong type for wmf..mediawiki_history - https://phabricator.wikimedia.org/T261261 (10SNowick_WMF) [01:57:26] 10Analytics: [BUG] Presto Error: parquet file declares column as wrong type for wmf..mediawiki_history - https://phabricator.wikimedia.org/T261261 (10SNowick_WMF) [05:58:41] 10Analytics, 10Analytics-Kanban: Check home/HDFS leftovers of drossi/fsalutari - https://phabricator.wikimedia.org/T258788 (10elukey) 05Open→03Resolved Everything cleaned up on stat100x for drossi and fsalutari. [06:02:37] 10Analytics: Check home/HDFS leftovers of demon - https://phabricator.wikimedia.org/T259585 (10elukey) 05Open→03Resolved All cleaned up! [06:02:58] 10Analytics, 10Analytics-Kanban: Check home/HDFS leftovers of lulu - https://phabricator.wikimedia.org/T261089 (10elukey) 05Open→03Resolved Removed the empty directories on stat100x hosts, closing.. [06:38:16] Hi team [06:41:47] bonjour! [06:59:34] (03PS1) 10Joal: Add page-artificial-id to mediawiki-history hive schema [analytics/refinery] - 10https://gerrit.wikimedia.org/r/622474 (https://phabricator.wikimedia.org/T261261) [07:02:16] (03CR) 10Joal: [V: 03+2 C: 03+2] "Fixed in prod - merging for consistency" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/622474 (https://phabricator.wikimedia.org/T261261) (owner: 10Joal) [07:08:09] 10Analytics, 10Patch-For-Review: [BUG] Presto Error: parquet file declares column as wrong type for wmf..mediawiki_history - https://phabricator.wikimedia.org/T261261 (10JAllemandou) Indeed it was a bug! We on purpose left out artificial-page-id field from the mediawiki-history hive schema. The field allows to... [07:08:26] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: [BUG] Presto Error: parquet file declares column as wrong type for wmf..mediawiki_history - https://phabricator.wikimedia.org/T261261 (10JAllemandou) a:03JAllemandou [07:28:55] 10Analytics-Clusters, 10Operations, 10decommission-hardware, 10ops-eqiad: Decommission analytics10[28-31,33-41] - https://phabricator.wikimedia.org/T227485 (10elukey) Updating this task - we are setting up the new hadoop test cluster, once done I'll clear all puppet config and set this task as actionable. [08:07:07] joal: o/ [08:07:19] heya elukey :) [08:07:45] for https://phabricator.wikimedia.org/T215858 we'll need to wipe labsdb1012 and transform it to a multi-instance set up.. not now but probably next Q [08:08:24] should be really easy to do, the downtime requested is around 1/2 days max to re-import the data, so easy to do between to sqoop runs [08:08:36] sounds perfect elukey [08:08:47] but sqoop will need to be aware of the new set up of course [08:10:57] elukey: Now that I have read the task - We exactly do what is described in the analytics-case of the task for comments: we read data from prod and sanitize it ouselves using needed fields - In fact this task shouldn't impact us as we already get comment and actor data from prod [08:14:19] joal: but sqoop assumes that all databases are available on port 3306 when getting data from labsdb1012 no? [08:14:36] in this case we'll need to use different ports based on what wiki is needed [08:14:44] (at least this is my understanding) [08:15:12] elukey: I have not understodd that from the task :( [08:16:07] joal: part of that giant project is to avoid the single-multi-source mariadb instance anymore, like our dear labsdb1012 is (since it is a nightmare to maintain) [08:16:34] elukey: can we chat in da cave for a moment? [08:16:48] joal: yes sure [08:43:42] Going AFK for about 1h [09:39:15] 10Analytics: Review current usage of HDFS and establish what/if data can be dropped periodically - https://phabricator.wikimedia.org/T261283 (10elukey) [09:43:28] 10Analytics: Review current usage of HDFS and establish what/if data can be dropped periodically - https://phabricator.wikimedia.org/T261283 (10elukey) [09:58:59] Back [10:29:49] * elukey lunch! (afk for a couple of hours) [11:33:28] (03PS4) 10Joal: Fix convertToSchema to work with array of structs [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/619034 (https://phabricator.wikimedia.org/T259924) (owner: 10Ottomata) [11:33:55] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: HiveExtensions.convertToSchema does not properly convert arrays of structs - https://phabricator.wikimedia.org/T259924 (10JAllemandou) a:05Ottomata→03JAllemandou [11:34:26] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: HiveExtensions.convertToSchema does not properly convert arrays of structs - https://phabricator.wikimedia.org/T259924 (10JAllemandou) >>! In T259924#6376540, @EBernhardson wrote: > Can you use spark higher order functions, particularly transform(array,... [12:10:12] (03PS3) 10Mforns: Adapt pingback queries to use the pingback heartbeat [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/621552 (https://phabricator.wikimedia.org/T246154) [12:17:32] (03CR) 10Mforns: [V: 03+2 C: 03+2] "Self-merging to let reportupdater backfill graphs and unbreak production." [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/621552 (https://phabricator.wikimedia.org/T246154) (owner: 10Mforns) [12:21:16] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10Platform Team Workboards (Initiatives): reportupdater Pingback reports are broken and need to be refactored - https://phabricator.wikimedia.org/T246154 (10mforns) I merged the patch with the changes in how we calculate the metric. This will soon popula... [12:27:37] heya elukey :] I'm trying to fix the pingback reportupdater reports, I have merged a new version of the queries, and now I'm making sure they get executed by reportupdater. [12:28:26] elukey: I'm missing to understand though what is happening... the logs for that job are stuck since June 26 [12:29:22] but the systemd timer is active and running [12:29:40] and the puppet job is ensure => present [12:34:20] oh, now I see the job is running since June 26 and the log shows reportupdater is stuck since then... [12:34:26] cool. [12:34:37] then restarting the timer should fix it, tryingf [12:39:32] I don't have permits to restart the reportupdater-pingback timer [12:39:56] but I killed the stuck reportupdater job [12:40:54] it's running! :] [12:47:18] here I am sorry [12:47:43] mforns: good! what command did you use to restart the timer? [12:47:50] you should have permits [12:52:34] (brb coffee) [12:55:23] 10Analytics-Radar, 10Better Use Of Data, 10Product-Analytics, 10Product-Infrastructure-Data, and 3 others: Session Length Metric. Web implementation - https://phabricator.wikimedia.org/T248987 (10mpopov) [13:23:17] ottomata: o/ [13:23:22] gooood morning [13:23:45] if you have time can we check the new buster schema vms and add them to LVS in case? [13:23:53] for https://phabricator.wikimedia.org/T255026 [13:23:54] (03CR) 10Ottomata: [C: 03+1] "Very nice!" (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/619034 (https://phabricator.wikimedia.org/T259924) (owner: 10Ottomata) [13:24:04] HellLo [13:24:33] elukey k checking [13:29:45] elukey: looks good to me! [13:29:46] sudo cumin 'schema*' 'curl http://localhost:8190/repositories/' [13:30:10] OH i can do that without cumin [13:30:17] (just couldn't from stat1004?) [13:31:13] ah yes probably schema* ips are not whitelisted without the proxy [13:31:21] for h in schema1003.eqiad.wmnet schema1004.eqiad.wmnet schema2003.codfw.wmnet schema2004.codfw.wmnet; do curl http://$h:8190/repositories/; done [13:31:22] ya [13:31:56] do they need to be whitelisted for the vlan? [13:33:37] ah no we whitelist the LVS IPs [13:33:38] okok [13:36:10] ottomata: created https://gerrit.wikimedia.org/r/622567 [13:39:19] following up with traffic for the patch [13:40:04] elukey: lgtm [13:40:23] cdanis: <3 [13:40:35] no pybal restart needed for just adding/removing nodes ofc [13:40:43] yes this is my understanding [13:41:18] they'll be added to the pool with state "pooled=no" IIRC [13:41:52] I would double-check that right after you puppet-merge, but I think so [13:42:00] might also be pooled=inactive [13:42:48] yes right [14:11:46] ottomata: ok to merge? [14:13:23] yes! [14:13:52] elukey: I used sudo -u analytics systemctl restart reportupdater-pingback (and I tried with .timer and also .server at the end) [14:14:11] mforns: you need to use only "sudo" [14:14:26] oh [14:14:38] ok :] [14:14:51] if the docs are not correct let's fix those [14:28:25] 10Analytics, 10Platform Team Sprints Board (Sprint 2), 10Platform Team Workboards (Green): Ingest api-gateway.request events to turnillo - https://phabricator.wikimedia.org/T261002 (10Pchelolo) a:03eprodromou [14:30:45] ottomata: so [14:30:46] {"schema1004.eqiad.wmnet": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=eventschemas,service=eventschemas"} [14:30:58] {"schema1003.eqiad.wmnet": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=eventschemas,service=eventschemas"} [14:31:22] I am going to set weight=10 and pooled=yes [14:34:33] ok all new vms added [14:34:38] a-team I'm at the dermatologist so I might not make it to standup, will send an e-scrum as soon as I'm back [14:34:41] https://config-master.wikimedia.org/pybal/eqiad/schema [14:34:54] https://config-master.wikimedia.org/pybal/codfw/schema [14:34:55] ack fdans [14:35:24] elukey: sounds good! [14:35:57] 10Analytics-Clusters, 10Patch-For-Review: Upgrade schema[12]00[12] to Debian Buster - https://phabricator.wikimedia.org/T255026 (10elukey) New vms added to the LVS endpoints: https://config-master.wikimedia.org/pybal/eqiad/schema https://config-master.wikimedia.org/pybal/codfw/schema If everything works corr... [14:36:18] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade schema[12]00[12] to Debian Buster - https://phabricator.wikimedia.org/T255026 (10elukey) [14:37:00] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade schema[12]00[12] to Debian Buster - https://phabricator.wikimedia.org/T255026 (10elukey) p:05Triage→03Medium a:03elukey [14:41:36] ottomata: so we could set the old vms with pooled=no, do a quick test and possibly merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/622587 ? [14:41:58] (even pooled=inactive:weight=0) [14:42:43] +1 [14:42:44] ottomata: about refine and structs - shall we plan on moving to spark 3 soon? [14:46:09] ottomata: all right lvs is serving only from newer vms (lemme know if the output is ok) [14:46:56] joal: coff coff camus coff coff bigtop coff coff [14:47:01] :D [14:47:07] hehe elukey :) [14:47:09] looks good to me elukey [14:47:11] I hear that [14:47:24] joal: i think we should consider it ya [14:47:41] yeah, maybe the q after we get to bigtop? [14:47:56] ottomata: I'm currently thinking of adding a UDF, but it's not trivial [14:48:03] yeah [14:48:11] i mean, it isn't ssuuuper urgent [14:48:14] no one has complained [14:48:20] it kinda sucks we don't have the $schema in hive [14:48:26] might be useful in the future [14:48:30] but afaik nothing is using it yet [15:01:02] ping ottomata razzi [15:01:12] ping fdans [15:01:57] ping razzi standdupp [15:25:21] 10Analytics, 10Analytics-Kanban: [BUG] Presto Error: parquet file declares column as wrong type for wmf..mediawiki_history - https://phabricator.wikimedia.org/T261261 (10Nuria) @elukey to update the setting in presto that assumes order on columns on hive metastore when looking fields up on parquet as this is l... [15:28:25] 10Analytics: Add urlshortener button to Turnilo - https://phabricator.wikimedia.org/T233336 (10Milimetric) You're the best, Luca. I tried this in the config and it worked locally: customization: urlShortener: | const shortner = 'https://meta.wikimedia.org/w/api.php?action=shortenurl&format=json';... [15:32:03] elukey: razzi https://meet.google.com/asb-opea-yig [15:39:14] elukey: ^ url shortner thing, lemme know if it's too hacky [15:39:30] (I can pass in the host from puppet or something?) [16:08:46] joal: the hive.parquet setting is on all the presto workers [16:09:07] elukey: with the restart? [16:09:15] elukey: you're TOO FAST :) [16:09:16] I didn't add it to the presto query/coordinator yet, in theory it seems not needed, but I can follow up if it doesn't wokr [16:09:21] yep restarted :) [16:09:40] * joal bows three times in respect [16:09:59] milimetric: niceeeee! We can test this on the staging vm (an-too1005), but there is an extra caveat - http calls need to go through the proxy :( [16:12:06] Ah right. But at least that’s on our side and doesn’t need upstream [16:12:28] I’ll take a look at proxy options for that request module [16:13:41] super [16:14:19] hm, doesn't look super useful (https://github.com/request/request#proxies) [16:17:45] elukey: what does "export https_proxy=http://webproxy.eqiad.wmnet:8080" do for CURL? It uses that as a proxy? [16:18:03] milimetric: exactly yes [16:18:16] oh so you can't connect directly to https://api-rw.discovery.wmnet/w/api.php [16:18:36] from inside the VLAN no [16:18:52] k, so I gotta somehow try that from the request module on an-tool1005 (is turnilo already deployed there?) [16:19:00] it is yes [16:19:25] ssh an-tool1005.eqiad.wmnet -L 9091:an-tool1005.eqiad.wmnet:9091 [16:36:15] 10Analytics, 10Product-Analytics, 10Structured Data Engineering, 10SDAW-MediaSearch (MediaSearch-Beta), 10Structured-Data-Backlog (Current Work): [L] Instrument MediaSearch results page - https://phabricator.wikimedia.org/T258183 (10CBogen) [16:37:33] hm, elukey: 404: Domain not configured\n

This domain points to a Wikimedia Foundation server, but is not configured on this server. [16:37:46] ... what? [16:42:06] milimetric: ah yes you didn't add the Host: meta.wikimedia.org header [16:42:23] if you want we can point to meta.wikimedia.org directly [17:04:52] milimetric: I tried to apply the change on an-tool1005, seems failing for syntax error :( [17:05:10] of course not really straightforward to get where is the error [17:28:33] joal: yt still? [17:50:37] ottomata: gone almost [17:50:49] ottomata: tmorrow morning? [17:51:11] going offline! o/ [17:51:18] Bye elukey [17:51:20] o/ [17:51:50] joal: sure! am trying to write a scala main class that uses java stuff without spark...for the first i'm i'm realizing [17:52:01] and i'm not totally sure the proper way to launch it and get classpath things right! :o [17:52:04] but we can talk tomorrow ya [17:57:51] joal: can you point me to the refinery-source patch that needs merging? :] [18:02:22] ah sorry freaked out forgot to do something for tech-com [18:02:30] I'll work on the thing Luca, thanks! [18:05:46] ah joal: I can see it in the etherpad, thanks! [18:09:02] back in a bit, dr appt [18:13:02] 10Analytics, 10Patch-For-Review: Add urlshortener button to Turnilo - https://phabricator.wikimedia.org/T233336 (10Milimetric) Ok, I'm running into the same `Blocked users can't make short URLs.` that Luca ran into above. Is that some kind of config on meta/the shortner that anyone knows about? [18:45:49] 10Analytics, 10Patch-For-Review: Add urlshortener button to Turnilo - https://phabricator.wikimedia.org/T233336 (10RhinosF1) Does anyone know what the external IP of the proxy is? I mean you could try authenticating to an account first? [19:15:25] 10Analytics, 10Analytics-Kanban, 10Platform Team Workboards (Initiatives): reportupdater Pingback reports are broken and need to be refactored - https://phabricator.wikimedia.org/T246154 (10CCicalese_WMF) I'm not sure if the reports are finished updating, but what is there looks promising. It would be good,... [20:33:35] 10Analytics-Radar, 10Performance-Team, 10MW-1.36-notes (1.36.0-wmf.7; 2020-09-01): Invalid navigation timing events - https://phabricator.wikimedia.org/T254606 (10Milimetric) >>! In T254606#6409428, @Gilles wrote: > ...is there a way for us to track a metric of how often these schemas are now hitting their m... [21:18:01] 10Analytics, 10Event-Platform: Allow analytics VLAN to reach all eventgate instances in all DCs - https://phabricator.wikimedia.org/T261356 (10Ottomata) [22:08:55] * nuria just resurrected her scala plugging for intelij that had MYSTERIOUSLY DIED [23:03:40] 10Analytics: Use types in Analytics Puppet classes/profiles/etc.. - https://phabricator.wikimedia.org/T252617 (10Nuria) Ping @razzi test test [23:04:22] 10Analytics, 10Analytics-Kanban: Use types in Analytics Puppet classes/profiles/etc.. - https://phabricator.wikimedia.org/T252617 (10Nuria) a:03razzi