[05:50:40] goood morning [07:18:44] 10Analytics, 10Operations, 10Traffic: Compare logs produced by atskfafka with those produced by varnishkafka - https://phabricator.wikimedia.org/T254317 (10ema) [07:18:52] 10Analytics, 10Operations, 10Traffic: Compare logs produced by atskfafka with those produced by varnishkafka - https://phabricator.wikimedia.org/T254317 (10ema) p:05Triage→03Medium [07:19:09] 10Analytics, 10Operations, 10Traffic: Compare logs produced by atskfafka with those produced by varnishkafka - https://phabricator.wikimedia.org/T254317 (10ema) [07:22:53] 10Analytics, 10Operations, 10Traffic: Compare logs produced by atskfafka with those produced by varnishkafka - https://phabricator.wikimedia.org/T254317 (10ema) [07:35:59] 10Analytics, 10Analytics-Kanban: Order mediawiki_history dumps by event_timestamp - https://phabricator.wikimedia.org/T254233 (10JAllemandou) The [[ https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/Dataset.html#sortWithinPartitions-org.apache.spark.sql.Column...- | `sortWithinPartition` ]] fu... [08:21:37] going to reimage druid1002 to Buster! [08:50:05] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Upgrade Druid to Debian Buster - https://phabricator.wikimedia.org/T253980 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts: ` druid1002.eqiad.wmnet ` The log can be found in `/var/log/wmf-auto-reimag... [08:50:18] !log reimage druid1002 to Buster [08:50:20] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:26:16] 10Analytics, 10Analytics-Kanban: Upgrade Druid to Debian Buster - https://phabricator.wikimedia.org/T253980 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['druid1002.eqiad.wmnet'] ` Of which those **FAILED**: ` ['druid1002.eqiad.wmnet'] ` [09:32:46] druid1002 up! [09:33:45] the coordinator sees it, and it is shuffling segments as expected [09:33:48] zookeeper cluster ok [09:34:10] last one standing for the analytics cluster is druid1003, then we'll move to the public cluster [09:35:36] !log re-run webrequest-druid-hourly-coord 03/06T7 (failed due to druid1002 moving to buster) [09:35:38] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:40:19] 10Analytics, 10Operations, 10netops: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10faidon) [10:17:06] 10Analytics, 10Operations, 10Traffic: Compare logs produced by atskfafka with those produced by varnishkafka - https://phabricator.wikimedia.org/T254317 (10elukey) Some extra context: Ema added prometheus monitoring for ATSKafka in https://grafana.wikimedia.org/d/1EUhPpzMz/atskafka?orgId=1, and the cp3050's... [10:23:34] * elukey lunch + errand! [10:42:43] a-team sorry for the alarms related to dumps, this is me backfilling [11:03:05] hi team [11:03:27] yay, a new druid host reimaged :) And some backfilling :) [11:04:45] o/ joal I hear you did some research into wdqs queries and stuff, is there a link to it somewhere? :D [11:05:23] Hi addshore [11:05:43] addshore: I did analysis in a notebook, and in scala-scripts [11:06:00] addshore: so no real link - I have a task to actually mjake some writing of that [11:06:45] addshore: I can spend some time showing you what I have if you wish [11:07:38] that could be great, as we have a workshop tommorrow where it might be useful! [11:07:53] sounds good - do you have time now? [11:08:06] I will have 20 mins in roughly 2 mins time! [11:08:40] 20mins is short given the stuff I have - I'll be quick :) [11:08:59] addshore: https://meet.google.com/rxb-bjxn-nip [11:09:05] :D [11:10:04] mforns: sqoop done, we can manually bump imagelinks table :) [11:12:05] poke joal, im in the waiting room [11:12:56] wowo [11:13:59] lag on my side [11:49:29] 10Analytics, 10Operations, 10netops: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10JAllemandou) Some more on the Druid aspect of things: - We have used multi-value dimensions in Druid without problem - Data needs to be an array and that's it. - We h... [12:05:57] joal: noted (sqoop) will do [12:06:07] thanks mforns :) [12:12:52] 10Analytics, 10Analytics-Kanban: Order mediawiki_history dumps by event_timestamp - https://phabricator.wikimedia.org/T254233 (10mforns) <3 [12:19:53] addshore: can you access that? https://docs.google.com/spreadsheets/d/e/2PACX-1vTTXM6TM4H-oDAx3R75p4x0CKyQR9SnZwwOeZ5VpTVi03KQYTmgJPRDyn4XN_uWIosrILzYA3RMq7W4/pubchart?oid=1755041942&format=interactive [12:22:14] also addshore, 1 modif about the numbers I gave: 2.4% of requests represent 76% of total query-time, those are requests taking more than 1s - Requests taking more than 10s represent 0.76% of all requests, and take 61.7% of time [12:34:22] joal: any reason not to close https://phabricator.wikimedia.org/T253753 ? [12:34:34] do you have a process after moving this to the [done] column? [12:34:53] gehel: we usually let nuria close them, as she prefers to have a view on everything [12:35:03] we do the same :) [12:35:08] ok, I'll leave it open [12:35:08] :) [12:35:12] thanks gehel [12:52:57] fdans: how dare you spamming us! :D [12:54:27] elukey: well I'm sooooo sorry I tarnished your spotless inbox luca [13:32:14] joal: yes I can access it thinks! [13:32:17] *thanks [14:33:03] (03CR) 10Joal: "A bunch of comments - I hope they make sense :)" (0314 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/597541 (https://phabricator.wikimedia.org/T252857) (owner: 10Fdans) [14:35:48] (03CR) 10Joal: "Forgot one comment - To follow the example of cassandra, let's add a `historical` folder in `pageview` folder, and inside that one add the" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/597541 (https://phabricator.wikimedia.org/T252857) (owner: 10Fdans) [14:37:01] fdans: hey [14:37:11] o/ [14:42:00] bearloga: o/ packages on stat boxes, will also check your product analytics user patch later on, really sorry but these are busy days :( [14:42:01] elukey: thank you! [14:42:35] elukey: no rush on that one :) i understand [14:46:10] fdans: ok, I'm lost, why was this reverted: https://github.com/wikimedia/analytics-wikistats2/commit/0a7f134e7693078e06b5a8984b9bc92c2b95a72a#diff-77a67180803f4123da260d41d18937bcL199-L203 [14:46:33] I vaguely remember you said something at the time but I read the gerrit and phab and didn't find any reasoning [14:47:04] putting that logic back in made the bug re-appear, and I still don't see where it breaks if you take it back out [14:47:39] milimetric: reading, remembering [14:47:55] fdans: take your time, it took me like two days [14:49:59] 10Analytics, 10Analytics-Kanban, 10Wikidata, 10Wikidata-Query-Service: Increase retention for mediawiki.revision-create on the kafka jumbo cluster - https://phabricator.wikimedia.org/T253753 (10Nuria) Given that retention is not on puppet is this a setting that is communicated to a new node when it joins t... [14:50:16] milimetric: ugh the truncated values thing [14:50:19] 10Analytics, 10Analytics-Kanban, 10Wikidata, 10Wikidata-Query-Service: Increase retention for mediawiki.revision-create on the kafka jumbo cluster - https://phabricator.wikimedia.org/T253753 (10Nuria) 05Open→03Resolved [14:53:02] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: EventGate validation errors should be visible in logstash - https://phabricator.wikimedia.org/T116719 (10Nuria) Does this have throtling of some sort? cause i can see how a high volume/defective schema could produce... [14:53:36] fdans: yeah, basically this.timeRange is observable and ties back to the $store. So setting that will mutate global state. Which is a little crazy and I'm sure we should be doing something else there. The Vue 3 stuff is much more explicit about observables, so maybe it'd be a good idea to upgrade and migrate just this part [14:53:52] 10Analytics, 10Analytics-Kanban: Fix oozie event dataset file - https://phabricator.wikimedia.org/T253855 (10Nuria) [14:55:20] milimetric: yea but I don't see how that ties to the code you just passed me [14:55:32] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: EventGate validation errors should be visible in logstash - https://phabricator.wikimedia.org/T116719 (10Ottomata) Nope, right now it is just error logs that eventgate logs anyway, so this would also be try of ANY ht... [14:56:18] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform: eventgate-wikimedia should expose runtime stream configuration - https://phabricator.wikimedia.org/T253157 (10Nuria) Should the path https://schema.wikimedia.org/stream-configs be available then? [14:56:50] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: EventGate validation errors should be visible in logstash - https://phabricator.wikimedia.org/T116719 (10Ottomata) Well, no that's not entirely true. Logstash is consuming from Kafka, so if there are too many messag... [14:57:19] 10Analytics, 10Analytics-Kanban: Move the Analytics infrastructure to Debian Buster - https://phabricator.wikimedia.org/T234629 (10elukey) [14:57:32] 10Analytics, 10Analytics-Kanban: Analytics Ops Technical Debt - https://phabricator.wikimedia.org/T240437 (10Nuria) [14:57:34] 10Analytics, 10Analytics-Kanban: Unify puppet roles for stat and notebook hosts - https://phabricator.wikimedia.org/T243934 (10Nuria) 05Open→03Resolved [14:57:38] fdans: early cave? [14:57:46] heh, like 30 seconds early [14:57:47] ok! [14:58:36] 10Analytics, 10Analytics-Kanban: Move the Analytics infrastructure to Debian Buster - https://phabricator.wikimedia.org/T234629 (10elukey) [15:00:16] 10Analytics, 10Analytics-Kanban: Upgrade matomo to the latest upstream - https://phabricator.wikimedia.org/T252741 (10Nuria) 05Open→03Resolved [15:00:18] 10Analytics, 10Analytics-Kanban: Move Matomo to Debian Buster - https://phabricator.wikimedia.org/T252740 (10Nuria) [15:03:18] 10Analytics, 10Patch-For-Review, 10Product-Analytics (Kanban): Need libfontconfig1-dev on stat hosts for data visualization work - https://phabricator.wikimedia.org/T254278 (10mpopov) Everything went fine installing systemfonts (after patch), gdtools, and hrbrthemes on stat1005, but we're missing libcairo2-d... [15:05:46] 10Analytics, 10Patch-For-Review, 10Product-Analytics (Kanban): Need some dependencies on stat hosts for data visualization work - https://phabricator.wikimedia.org/T254278 (10mpopov) [15:10:00] 10Analytics, 10Analytics-Kanban: Move the Analytics infrastructure to Debian Buster - https://phabricator.wikimedia.org/T234629 (10elukey) [15:12:12] elukey: thanks again! :) [15:12:50] running puppet on stat1008 now! [15:16:01] bearloga: done! [15:16:12] elukey: appreciate it, thank you! [15:17:55] 10Analytics, 10Patch-For-Review, 10Product-Analytics (Kanban): Need some dependencies on stat hosts for data visualization work - https://phabricator.wikimedia.org/T254278 (10mpopov) 05Open→03Resolved Done! @nettrom_WMF: all good to go on stat1005 and stat1008! On other hosts the older version (0.6.0) ne... [15:21:45] djellel: snapshot 2020-04 up-to-date in mediawiki_wikitext_history :) [15:22:50] for all langs ? [15:31:51] yes djellel [15:35:47] elukey https://meet.google.com/dcd-vvqb-dhd [15:35:50] joal: superb, thank you :) [15:47:01] 10Analytics, 10Analytics-Kanban: Add page_restrictions table to hive - https://phabricator.wikimedia.org/T253803 (10Nuria) 05Open→03Resolved [15:47:34] mforns, joal : have any of you looked at mediawiki history scooping? [15:47:55] 10Analytics, 10Analytics-Kanban: Add page_restrictions table to hive - https://phabricator.wikimedia.org/T253803 (10Nuria) [15:48:39] 10Analytics, 10Analytics-Kanban: Update sqoop before labs views change - https://phabricator.wikimedia.org/T252565 (10Nuria) 05Open→03Resolved [15:49:22] 10Analytics, 10Analytics-Kanban, 10Research, 10Patch-For-Review: Proposed adjustment to wmf.wikidata_item_page_link to better handle page moves - https://phabricator.wikimedia.org/T249773 (10Nuria) [15:49:31] 10Analytics, 10Analytics-Kanban, 10Research, 10Patch-For-Review: Proposed adjustment to wmf.wikidata_item_page_link to better handle page moves - https://phabricator.wikimedia.org/T249773 (10Nuria) 05Open→03Resolved [15:50:37] 10Analytics, 10Analytics-Kanban: Spike, see how easy/hard is to scoop all tables from Eventlogging log database - https://phabricator.wikimedia.org/T250709 (10Nuria) this task is not completed, correct, have we scooped all tables cc @Milimetric [15:50:43] 10Analytics, 10Operations, 10Product-Analytics, 10SRE-Access-Requests: Hive access for Sam Patton - https://phabricator.wikimedia.org/T248097 (10spatton) @Dzahn, @jcrespo, @mpopov, @nettrom_WMF, @Nuria, @Volans : thank you for the pursuit, and sorry it was necessary. Busy times :) I really appreciate your... [15:51:00] 10Analytics: Analytics Hardware for Fiscal Year 2019/2020 - https://phabricator.wikimedia.org/T244211 (10Nuria) [15:51:01] 10Analytics, 10Analytics-Kanban: Add new Druid nodes to analytics and public clusters - https://phabricator.wikimedia.org/T252771 (10Nuria) 05Open→03Resolved [15:51:26] 10Analytics-Kanban, 10Better Use Of Data, 10Product-Analytics, 10Patch-For-Review: Upgrade to Superset 0.36.0 - https://phabricator.wikimedia.org/T249495 (10Nuria) 05Open→03Resolved [15:51:28] 10Analytics-Kanban, 10Better Use Of Data, 10Product-Analytics: Superset Updates - https://phabricator.wikimedia.org/T211706 (10Nuria) [15:51:40] nuria: you mean the msck repair of mediawiki_imagelinks? [15:51:54] 10Analytics, 10Analytics-Kanban: Camus failing to import eqiad.mediawiki.(api|cirrussearch)-request from partitions leaders on kafka-jumbo1006 - https://phabricator.wikimedia.org/T252203 (10Nuria) 05Open→03Resolved [15:52:54] 10Analytics, 10Analytics-Kanban: Fix oozie event dataset file - https://phabricator.wikimedia.org/T253855 (10Nuria) 05Open→03Resolved [15:52:57] nuria: joseph pinged me when sqoop was done, and I just repaired the table, so mediawiki_imagelinks should have the full 2020-05 snapshot now [15:53:10] 10Analytics, 10Analytics-Kanban: Add page_restrictions table to sqoop list - https://phabricator.wikimedia.org/T251749 (10Nuria) 05Open→03Resolved [15:54:19] mforns: no, i mean the mw history overall, for example, i am not clear whether the page_resrictions table that we just have added to scoop (1st time this snapshot) was scooped or not [15:55:09] mforns: ah ya, it is there, so we did include it [15:55:17] ok [15:56:07] 10Analytics, 10Analytics-Kanban: Update oozie SLAs for pageview-daily-dumps and wikidata-entities jobs - https://phabricator.wikimedia.org/T253847 (10Nuria) 05Open→03Resolved [15:56:16] 10Analytics, 10Analytics-Kanban: Update oozie SLAs for pageview-daily-dumps and wikidata-entities jobs - https://phabricator.wikimedia.org/T253847 (10Nuria) [15:56:33] BTW miriam, the mediawiki_imagelinks table has now data for the latest snapshot, and will be updated monthly [15:58:08] oh that's great, thanks so much mforns and all :) [15:58:27] no problemo! [16:00:03] mforns: I was thinking that if you need to test airflow on a more powerful hw, I can re-deploy the analytics keytab on all the stat boxes [16:00:43] originally me and joal thought it was a good idea security wise but it is showing some trade-offs for us [16:01:00] also joal, sqoop finished?? [16:01:06] shall I reboot the launcher? [16:01:21] (stopping timers first and letting them drain of course!) [16:01:25] elukey: hm, not sure more hardware would be necessary? but however you see better [16:01:34] elukey: Forgot to tell you ! [16:01:39] elukey: sqoop finished, yes [16:01:44] mforns: well on an-launcher it was consuming a lot of CPU sadly :( [16:01:57] !log stop timers on an-launcher, prep for reboot [16:01:58] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:02:00] elukey: that is weird! [16:02:21] 10Analytics, 10Operations, 10Product-Analytics, 10SRE-Access-Requests: Hive access for Sam Patton - https://phabricator.wikimedia.org/T248097 (10Nuria) Seems that your data access needs wouls be served by https://turnilo.wikimedia.org It does not seem you need access to raw data. Do you have an account on... [16:03:19] elukey: then, I haven't felt any slowness when testing, but if the test slowed down other production jobs, then sure, I'll move to a more beefy machine [16:03:32] see https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&refresh=5m&var-server=an-launcher1001&var-datasource=eqiad%20prometheus%2Fops&var-cluster=analytics&from=now-7d&to=now [16:04:04] when it goes up and up it was probably sqoop kicking in [16:04:12] but before that it was around 50% stable [16:04:41] (not horrible but a little concerning) [16:05:21] brb [16:13:36] nuria: one thing that I forgot - on superset staging I deployed the "async" coroutine engine, it seems working well so I am wondering if we should deploy it or not [16:17:19] 10Analytics: Decomission notebook hosts - https://phabricator.wikimedia.org/T249752 (10SNowick_WMF) I have moved all my files off of nb3 and nb4 [16:18:40] 10Analytics: Decomission notebook hosts - https://phabricator.wikimedia.org/T249752 (10elukey) This coming Friday I'll remove ssh access to the nodes and wait some other days before decommissioning the nodes. [16:36:26] 10Analytics, 10Operations, 10Traffic: missing wmf_netflow data, 18:30-19:00 May 31 - https://phabricator.wikimedia.org/T254161 (10elukey) The missing data seems from May 31st 18:30 to 19:00. I did a quick check via Spark and on HDFS the data seems present: ` scala> spark.sql("select stamp_inserted from wmf.... [16:39:58] mmm mforns do you have a min? [16:40:16] elukey: yes [16:40:24] thanks! [16:40:29] so I need to reboot an-launcher1001 [16:40:35] k [16:40:36] and I am waiting for all the timers to complete [16:40:44] (I stopped their recurrence) [16:40:57] two are still running, and are el2druid hourly/daily [16:40:58] aha [16:41:03] but daily is https://yarn.wikimedia.org/cluster/app/application_1589903254658_70383 [16:41:09] and it started 16h ago? [16:41:27] I see succeeded [16:41:44] lol it completed now [16:41:53] uou! [16:41:56] 16h??? [16:41:56] just invoking you ahhahaha [16:42:20] maybe because the cluster was busy with mediawiki denormalize until 10 mins ago [16:44:53] hourly is also taking ~5h https://yarn.wikimedia.org/cluster/app/application_1589903254658_74802 [16:45:32] wow [16:45:53] 1 hour is taking 5 hours to compute? [16:46:27] more specifically [16:46:28] --config_file eventlogging_to_druid_netflow_hourly.properties --since 2020-06-03T05:00:00 --until 2020-06-03T08:00:00 [16:46:28] this is shocking [16:47:05] ah yes both are netflow related [16:47:05] 10Analytics, 10Analytics-Kanban: Spike, see how easy/hard is to scoop all tables from Eventlogging log database - https://phabricator.wikimedia.org/T250709 (10Milimetric) No, sadly a few of them still failed and I'm rerunning them in the background. It's just two more Edit_* schemas and all the mediawiki_* ones. [16:47:34] is it doing three hours at a time? [16:48:23] apparently [16:48:53] the daily was doing [16:48:54] --config_file eventlogging_to_druid_netflow_daily.properties --since 2020-05-30T00:00:00 --until 2020-05-31T00:00:00 [16:51:28] I checked the yarn logs for the daily one but didn't find a lot [16:53:53] the cluster is at 90% usage, most of it from the default queue [16:55:05] there is a mforns guy using 1347584 MB of hadoop RAM :D [16:55:28] elukey: yes, I'm testing the mediawiki history dumps modification [16:55:54] I waited for the mediawiki history denormalized to finish, but still [16:55:59] I started like 20 mins ago [16:56:01] mforns: is it expected to consume 1/4th of the ram in the cluster? [16:56:06] (no big deal, just to know) [16:57:03] hmm, the last time I tested that job I used this settings, but not sure this still applies :/ [16:57:27] !log reboot an-launcher1001 to get new memory [16:57:28] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:58:00] it's approximately what the mediawiki history denormalized job was consuming, and it processes the same data size, so I guess makes sense [16:58:07] mforns: no problem for me Marcel, just keep an eye on it once in a while to see if it is stable or growing [16:58:13] ram is there to be used :) [16:58:30] elukey: I can retry with less executor-memory if you think [16:59:24] it shouldn't be long though... 1 more hour [16:59:57] mforns: nono please keep going, mine was just a suggestion to have an idea about resource consumption when we launch it, nothing more [17:00:11] aha [17:01:51] elukey: (saw your note about rsync) did we restarted the VM with more ram to be able to run reportupdater jobs? [17:02:11] nuria: I am doing it now :) [17:02:25] elukey: ah ya, SHOULD HAVE READ BACKCROLL! [17:02:30] *backscroll [17:02:57] and a reboot is not sufficient sigh [17:03:02] * elukey reads again the docs [17:05:48] team, I see two mediawiki-geoeditors-monthly-coords running at the same time, can I kill the one that failed? [17:07:02] +1 [17:08:56] ok it worked, an-launcher1001 has now 12g :) [17:09:42] \o/ [17:10:29] mforns: RU jobs restarted [17:10:33] :] [17:10:40] !log restart RU jobs after adding memory to an-launcher1001 [17:10:41] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:11:13] 10Analytics, 10Analytics-Kanban, 10Operations: Increase memory available for an-launcher1001 - https://phabricator.wikimedia.org/T254125 (10elukey) [17:31:09] 10Analytics: Investigate why netflow hive_to_druid job is so slow - https://phabricator.wikimedia.org/T254383 (10mforns) [17:39:09] joal: o/ [17:39:24] your spark shell seems quite big, 1180672 MB :( [17:39:46] true elukey :) Will release it in minutes [17:40:01] super, Marcel is using another TB so the cluster is super full [17:43:12] Done elukey [17:46:11] <3 [17:49:38] mforns: can I bother you another sec? (for a el2druid question) [17:57:07] (nevermind tomorrow :) [17:57:08] * elukey off! [18:04:17] :[ [18:04:45] sorry joal for cluster use [18:05:00] np mforns :) [19:21:22] 10Analytics, 10Analytics-EventLogging, 10EventStreams, 10Wikimedia-production-error: Argument 1 passed to MediaWiki\Extension\EventStreamConfig\ApiStreamConfigs::multiParamToAssocArray() must be of the type array, null given, called in /srv/mediawiki/php-1.35.0-wmf... - https://phabricator.wikimedia.org/T254390 [19:21:52] 10Analytics, 10Analytics-EventLogging, 10EventStreams, 10Wikimedia-production-error: Argument 1 passed to MediaWiki\Extension\EventStreamConfig\ApiStreamConfigs::multiParamToAssocArray() must be of the type array, null given, called in /srv/mediawiki/php-1.35.0-wmf... - https://phabricator.wikimedia.org/T254390 [19:22:02] 10Analytics, 10Analytics-EventLogging, 10EventStreams, 10Wikimedia-production-error: Argument 1 passed to MediaWiki\Extension\EventStreamConfig\ApiStreamConfigs::multiParamToAssocArray() must be of the type array, null given, called in /srv/mediawiki/php-1.35.0-wmf... - https://phabricator.wikimedia.org/T254390 [19:22:43] Hey everyone. New train blocker bug ^^ – massive rise in PHP fatals when I tried to roll out wmf.35 to group1 (i.e., meta). [19:24:40] ottomata: would you be somewhere around? [19:29:34] James_F: while I have noted your message, I can't help on the issue - I've pinged ottomata as the main owner of that thing [19:29:37] 10Analytics, 10Analytics-EventLogging, 10EventStreams, 10Wikimedia-production-error: Argument 1 passed to MediaWiki\Extension\EventStreamConfig\ApiStreamConfigs::multiParamToAssocArray() must be of the type array, null given, called in /srv/mediawiki/php-1.35.0-wmf... - https://phabricator.wikimedia.org/T254390 [19:29:39] mforns: by any chance? [19:29:47] joal: Understood. [19:29:58] yes, what? [19:30:24] hi mforns - IRRC you are more or less familiar with EventStream [19:30:36] mforns: there is a blocker in mediawiki-train realted to stream-config [19:30:42] ufa not really [19:30:47] https://meta.wikimedia.beta.wmflabs.org/w/api.php?format=json&action=streamconfigs&all_settings=true&streams=test.event throws on BetaCluster too. [19:31:39] ack James_F - we're gonna need ottomata, none of us here can help except him I think [19:31:45] uou, not sure what to do there [19:31:50] sorry for that James_F :( [19:31:58] No worries. :-) [19:37:03] ping again ottomata? [19:38:02] If one of you could review https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/EventStreamConfig/+/602143 quickly I think that'd fix it. [19:38:52] * joal has read PHP for the time since at least few years [19:38:57] * James_F grins. [19:39:02] My sympathies. [19:39:08] James_F: it feels ok to me, but I can't help more :) [19:39:34] The problem about typed languages is you have to actually make sure your type fits. ;-) [19:39:57] right - EXPLICIT TYPING :) [19:45:39] joal: do you have 5 mins to re-discuss RSVD sparsity? :D [19:46:02] mforns: In tech-dpt meeting [19:46:10] mforns: And once done I'll go to bed :) [19:46:17] tomorrow afternoon mforns ? [19:46:18] oh ok, no worries [19:46:22] sure! [19:46:28] :] [19:55:20] joal: you lisenting to tech dept meeting rn? [19:55:24] yup [19:55:28] have those guys talked to you? [19:55:32] or us? [19:55:40] ottomata: I don't think so :D [19:55:54] I was thinking about that as well [19:56:23] ottomata while you're here - can you scroll back and look at the train-blocker above please? [19:57:18] oh! [19:57:35] sorry James_F looking [19:58:05] Thanks ottomata :) [20:00:12] patching [20:01:06] ottomata: Have a patch written if that's OK. [20:01:12] ya am amending [20:02:42] elukey or anybody who knows: quick question: [20:03:09] we've accidentally connected some consumer-groups to topics they shouldn't have been subscribed to in jobqueue [20:03:33] so now burrow has these crazy metrics for topic-group pairs that are wrong [20:04:08] do you know if there's like some timeout, or grace period, after which these errorneous metrics will disappear from burrow? [20:06:19] having a vagrant issue... [20:06:41] We just call that "vagrant". :-) [20:06:53] Error: Class 'Wikimedia\IPUtils' not found in /vagrant/mediawiki/includes/WebRequest.php on line 269 [20:07:20] ottomata: Sounds like you need to run composer update. [20:07:50] i sometimes try vagrant git-update but it never suceeds [20:08:14] trying [20:39:16] James_F: ammended [20:40:18] ottomata: LGTM. Merge and I'll backport. [20:40:24] k [20:40:36] waiting for jenkins [20:40:58] Just C+2 it. [20:41:10] Jenkins won't let it be merged unless it passes, and you can go about your day. :-) [20:48:41] James_F thanks very much for your help [20:48:58] ottomata: Happy to help. [20:52:51] (03PS2) 10Ottomata: Refine - Make event transform functions smarter about choosing which possible column to use [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/601865 [21:09:12] (03CR) 10Ottomata: "Still testing, so WIP" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/601865 (owner: 10Ottomata) [21:53:09] 10Analytics, 10Analytics-EventLogging, 10EventStreams, 10Patch-For-Review, 10Wikimedia-production-error: Argument 1 passed to MediaWiki\Extension\EventStreamConfig\ApiStreamConfigs::multiParamToAssocArray() must be of the type array, null given, called in /srv/me... - https://phabricator.wikimedia.org/T254390 [22:54:20] o/ [22:54:49] am not able to rsync between 1006 and 1007 [22:57:26] djellel: tell me more :) [22:57:32] got a few mins real quick to help :) [22:57:57] ottomata: trying to move a file between the two machines [22:58:38] I experienced a similar problem before, but I thought the ports were opened since. [22:59:45] ottomata: and don't worry, it's not urgent, just fyi :)