[00:18:58] 10Analytics, 10Analytics-Kanban, 10DBA, 10Data-Services, 10Core Platform Team Backlog (Watching / External): Not able to scoop comment table in labs for mediawiki reconstruction process - https://phabricator.wikimedia.org/T209031 (10CCicalese_WMF) [00:19:16] 10Analytics, 10EventBus, 10Operations, 10WMF-JobQueue, and 5 others: Kafka eqiad.mediawiki.page-delete topic is empty - https://phabricator.wikimedia.org/T210451 (10Pchelolo) Not to be worried. We have all the failed events stored since 2018-04-18. If needed, I will fetch all the missing page deletes tomor... [00:21:49] 10Analytics, 10EventBus, 10Operations, 10WMF-JobQueue, and 5 others: Kafka eqiad.mediawiki.page-delete topic is empty - https://phabricator.wikimedia.org/T210451 (10Ottomata) Oo, I just did the same, or, at least I copied the relevant files. They are on stat1004:/home/otto/eventbus-validation-logs0. Stas... [00:23:27] 10Analytics, 10EventBus, 10Operations, 10WMF-JobQueue, and 5 others: Kafka eqiad.mediawiki.page-delete topic is empty - https://phabricator.wikimedia.org/T210451 (10Smalyshev) I think I've extracted all I need from the DB tables for now, but I'll double-check and if anything is still missing I check the ex... [02:51:40] 10Analytics, 10Analytics-Data-Quality, 10Product-Analytics: mediawiki_history datasets have null user_text for IP edits - https://phabricator.wikimedia.org/T206883 (10Neil_P._Quinn_WMF) >>! In T206883#4757850, @JAllemandou wrote: > I hear your point and it makes a lot of sense. I think our views differ in th... [07:38:31] 10Analytics, 10Analytics-Kanban, 10DBA: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10elukey) p:05Triage>03High [07:39:28] hic sunt leones --^ [08:10:08] Hi elukey - Good hunting --^ [08:17:30] :D [08:17:35] bonjour! [08:17:42] Bonjour :) [08:18:21] so joal I was thiking about the Banner impression "real time" ingestion, and maintaining a spark streaming job only to add normalization [08:18:58] elukey: I think it'd do a bit more, like expanding maps possibly, but maybe I'm wrong :) [08:19:19] joal: yeah but what kind of maps? They don't need any at the moment [08:19:37] oh right - I thought we wanted to add geoloc for instance [08:22:43] my current thought is - what is the minimum amount of config that we need to make everything working since it is the 27th of Nov :D [08:22:46] ? [08:23:14] right - If we go without normalization, we can try an ingestor job [08:23:46] exactly, very quick and possibly ready with a couple of hours of work and some swearing :D [08:24:49] possibly a lot more swearing than anything else :) [08:25:39] elukey: do you want me to give it a try? [08:26:53] joal: if you have time yes, otherwise I was planning to spend some time on it later on today [08:35:01] joal: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/475964/1/modules/druid/templates/log4j2.xml.erb [08:35:39] elukey: meh ? [08:36:00] my fault, minor thing but now the logs are not split in two [08:36:45] last log in every -metrics.log was 2018-10-25T10:31:16.271Z :P [08:37:10] good to merge? [08:37:14] Ah ok - please go :) [08:37:26] I also have to roll restart all druid daemons for jvm upgrades [08:38:25] ok [08:41:33] that is the first with 0.12.3, hopefully quiet :D [08:44:57] Event [{"feed":"metrics","timestamp":"2018-11-27T08:44:52.517Z" [08:45:01] good now :) [08:45:08] (in -metrics.log) [08:45:17] so I am doing the historicals now on the private cluster [08:46:08] geeat elukey :) Thanks ! [08:46:39] elukey: have ou deleted the central-notice impression data? [08:47:37] joal: I did yes [08:47:41] (5 mins ago) [08:48:01] elukey: I have seen that - I was using it as a baseline for realtime config [08:48:38] joal: ah sorry! It was a test without the right dimensions to use, so I cleaned it up [08:48:44] I can give you the list [08:48:56] Is it documented in the task elukey ? [08:49:04] it should but I am not sure [08:49:27] we can make the final list in there [08:53:57] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Return to real time banner impressions in Druid - https://phabricator.wikimedia.org/T203669 (10elukey) List of dimensions that we are going to grab from the Eventlogging event: ` event_campaign,event_banner,event_project,event_uselang,event_bucket,event_anonym... [08:54:01] joal: --^ [08:54:37] elukey: Noted - Will use that :) Thanks ! [08:55:46] elukey: I'm assuming we're gonna keep wiki or webHost, and possibly some userAgent info? [08:56:41] elukey: or maybe event_db ? [08:56:58] joal: I don't think so, those are not in https://turnilo.wikimedia.org/#banner_activity_minutely right ? [08:57:58] elukey: true, but I'm assuming it's a mistake [08:58:17] project is project-family for us - and db hass the lang part [08:59:16] you know that I am ignorant about this part, I am only reporting what I am seeing in the current data that we have to replicate :) [08:59:24] :) [09:00:42] elukey: I'm gonna make the list what seems relevant in the event, and we'll discuss with them I assume [09:01:26] sure [09:01:44] if more complex fields are needed (like maps etc..) then a spark job makes sense [09:01:59] I am reluctant in maintaining a job if not needed [09:02:08] the last one (with tranquillity) was a bit of pain [09:02:13] elukey: I think we should be ok [09:02:46] "don't worry, about a thing, everything is gonna be all right.." [09:04:20] :D [09:14:37] druid private restart done [09:14:41] moving to druid public [09:27:44] so I am reading 0.13 release notes (still WIP) https://github.com/apache/incubator-druid/issues/6442 and they seem to be in the direction of making zookeeper optional [09:27:53] looks good :) [09:33:28] elukey: https://turnilo.wikimedia.org/#test_kafka_event_centralnoticeimpression [09:35:10] wooooooooooooooooooooooooooooooooooooowwwwwwwwwwwwwwwwwwwwwwwwww [09:35:52] elukey: monitoring currently to see how the thing behaves in term of tasks and all [09:35:53] even with UA map? [09:36:17] elukey: UA is json, I use a flattener spec, so I get what I want [09:36:27] very nice [09:36:41] elukey: there is redundancy is fields, but I put almost everything [09:36:57] for instance: wiki and event_db [09:37:26] and it only needs an indexation job right? [09:37:31] I put everything cause I think it could help sometimes, for debuging purposes for instance (recvFrom for instance) [09:38:07] elukey: supervision job - This means Druid manages by itself getting stuff from kafka (I think it uses tranquility behind the scene) [09:38:41] yeah but a supervision job is basically an indexation job with small batches no? [09:39:11] well it depends what you mean b indexation job [09:40:17] elukey: what we usually call indexation job happens in hadoop, with data being read from HDFS [09:40:39] elukey: in supervisor case the job is a realtime-indexation task, reading from kafka [09:41:03] elukey: you can tunnel to overlord (druid1002:8090) and look at the tasks :) [09:41:17] what I mean is turning data into segments, that eventually will be handed off to historicals [09:41:37] elukey: overlord makes no difference between indexation tasks, but I think we should always make a difference between batch and realtime ones [09:41:50] elukey: historical have some segments now [09:41:54] sure I'll keep it it mind [09:42:33] joal: yes IIUC they get the segments after the realtime indexation reaches a certain threshold [09:42:55] elukey: I configured realtime-tasks to last 10 minutes, that means we'll have segments of 10minutes in historical, but it's not big deal as we shall overwrite them with Marcel job [09:43:06] ack [09:43:23] elukey: segments are handed-off to historical once "finalized", meaning the task is done [09:43:46] elukey: I configured smaller time than 1h to try to prevent overflow tomorrow ;) [09:44:41] joal: the task is done after reaching the 10 minutes collection, then the segments are handed off to historicals [09:44:44] right? [09:45:07] correct ! [09:45:14] now I am wondering if we have metrics for the supervisor [09:45:17] or at least, that's my undersntading :) [09:45:18] probably yes [09:45:58] so first one is [09:45:59] Event [{"feed":"metrics","timestamp":"2018-11-27T09:43:48.717Z","service":"druid/overlord","host":"druid1002.eqiad.wmnet:8090","version":"0.12.3","metric":"ingest/kafka/lag","value":2,"dataSource":"test_kafka_event_centralnoticeimpression"}] [09:49:27] probably the druid/peon metrics that are emitted now in the middlermanager-metrics.log [09:50:26] ah! [09:50:27] druid_realtime_ingest_events_processed_count{datasource="test_kafka_event_centralnoticeimpression"} 1667.0 [09:50:31] \o/ [09:52:15] \o/ :) [09:52:17] Awesome :) [09:53:28] https://grafana.wikimedia.org/dashboard/db/druid?refresh=1m&panelId=41&fullscreen&orgId=1&var-datasource=eqiad%20prometheus%2Fanalytics&var-cluster=druid_analytics&var-druid_datasource=All&from=now-3h&to=now [09:53:34] didn't even think about it [09:53:43] 10Analytics, 10Operations, 10Performance-Team, 10Traffic: Only serve debug HTTP headers when x-wikimedia-debug is present - https://phabricator.wikimedia.org/T210484 (10Gilles) [09:55:13] elukey: Nice :) [09:55:24] elukey: wil kill and relaunch supervisor [10:00:26] elukey: would you mind restarting turnilo for me? I have a possibly successful test running on :) [10:02:18] joal: you should be able to restart it with sudo etc.. [10:02:37] let's try to see if it works (I don't think that anybody did it after the new perms were deployed) [10:02:52] elukey: no prob ! I need help on the machine though [10:03:04] mmm your user is not on analytics-tool1002 [10:03:52] lemme check, I was convinced otherwise [10:03:54] weird [10:04:13] ahhhhhh [10:04:20] analytics-admins is not deployed to those hosts [10:04:21] tried to connect to analytics-tool1002 - no good yet [10:07:04] creating a patch now [10:09:54] joal: can you try now? [10:09:58] sure [10:10:55] login ok, but can't sudo for sudo systemctl restart turnilo.service [10:11:03] I also tried sudo -u analytics - no chance [10:11:06] elukey: --^ [10:13:21] ah sorry, sudo -u turnilo systemctl restart turnilo [10:13:53] nope :( [10:14:06] not on analytics-tool1002? [10:14:17] The name org.freedesktop.PolicyKit1 was not provided by any .service files [10:15:27] ??? [10:15:45] oh yes I can repro [10:15:50] status does not return that [10:16:28] Right - status works for me as well [10:16:42] elukey: stop / stat? [10:16:46] elukey: stop / start? [10:16:49] or try-restart? [10:17:07] I think it is a perm issue [10:17:18] probably the sudoers rule is not the right one [10:17:34] :/ [10:17:54] because you have %analytics-admins ALL = (turnilo) NOPASSWD: ALL [10:18:09] and I thought that it was sufficient to restart a service [10:18:12] but maybe not [10:18:22] hm - /me is not sudoer-fluent :/ [10:19:10] joal: restarted turnilo in the meantime, so you are unblocked [10:19:13] really annoying :( [10:20:16] elukey: Have a look at https://turnilo.wikimedia.org/#test_kafka_event_centralnoticeimpression medtrics [10:20:23] \o/ !!!! [10:20:33] * joal sings and dances [10:20:48] sure [10:21:36] elukey: http://druid.io/docs/latest/ingestion/transform-spec.html [10:21:38] what??? normalized count? [10:21:45] :D [10:21:49] wow!! [10:21:58] this is really awesome [10:22:06] * joal is happy to have read the docs again [10:22:59] joal: can you add it to the task so everybody can check it (and possibly also start looking at metrics) [10:23:18] yes elukey - Will do [10:25:58] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Return to real time banner impressions in Druid - https://phabricator.wikimedia.org/T203669 (10JAllemandou) I have launched a realtime job indexing values flowing in kafka. Data can be seen here (please notice the event normalized count metric :) : https://tu... [10:26:01] elukey: --^ [10:28:32] We should ping mforns as well on that - There is a "bucket extraction function" defined in http://druid.io/docs/latest/querying/dimensionspecs.html [10:28:36] super! [10:28:41] elukey: --^ as well :) [10:29:28] elukey: maybe there is not even the need to modify spark code for EL2Druid to get buckets, inverts and all [10:29:38] mforns: --^ [10:31:54] but it needs to be added to the indexing specs right? So probably Marcel will need to modify EL2Druid anyway [10:33:12] joal: I filed https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/475984/1/modules/admin/data/data.yaml for the sudoers rule [10:33:28] you guys need to be able to restart stuff in case of an emergency [10:35:47] the new stream is so awesome, great work1 [10:35:48] !! [10:43:11] elukey: supervisor is really great :) [10:45:28] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Return to real time banner impressions in Druid - https://phabricator.wikimedia.org/T203669 (10JAllemandou) For reference, here is the request sent to druid for realtime ingestion: ` curl -L -X POST -H 'Content-Type: application/json' -d '{ "type": "kafka",... [10:59:20] heya team :] [10:59:41] joal, elukey reading [11:05:17] elukey, how did you manage to add the normalized count to turnilo?? [11:07:04] mforns: joal did it with http://druid.io/docs/latest/ingestion/transform-spec.html [11:07:17] but that is the real time ingestion from kafka [11:07:24] elukey, I see! [11:07:27] so druid reading directly from the topic [11:12:43] joal, the bucket extraction function looks useful, but for the case of time measures, we've been using an "exponential" bucketing (similar to orders of magnitude), this seems not supported by the extraction function. However, the time buckets seem not to be of real value to analysts, so... [12:19:18] 10Analytics, 10Analytics-Cluster: Upgrade Hive to ≥ 2.0 - https://phabricator.wikimedia.org/T203498 (10elukey) Keeping the task updated - in https://issues.apache.org/jira/browse/BIGTOP-3074 the BigTop Apache distribution removed the oozie packaging since it seems not compatible (yet) with Hive 2.x. The CDH6 d... [12:31:10] 10Analytics, 10Analytics-Wikimetrics, 10Patch-For-Review, 10WorkType-Maintenance: flake8 errors on wikimetrics - https://phabricator.wikimedia.org/T210320 (10rafidaslam) Okay, no problem. Just a note, if we wanted to fix both `W504 line break after binary operator` and `W503 line break before binary opera... [12:41:59] 10Analytics, 10EventBus, 10Operations, 10WMF-JobQueue, and 5 others: Kafka eqiad.mediawiki.page-delete topic is empty - https://phabricator.wikimedia.org/T210451 (10mobrovac) The fix has been deployed, delete events should start flowing again, so resolving. Let's reopen the ticket if that does not occur. [12:42:06] 10Analytics, 10EventBus, 10Operations, 10WMF-JobQueue, and 5 others: Kafka eqiad.mediawiki.page-delete topic is empty - https://phabricator.wikimedia.org/T210451 (10mobrovac) 05Open>03Resolved a:03mobrovac [13:01:23] hallo [13:01:30] about https://gerrit.wikimedia.org/r/#/c/analytics/limn-language-data/+/475618/1/cx/config.yaml [13:01:46] (03CR) 10Amire80: Add a scheduled job for daily CX abuse filters statistics (032 comments) [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/475618 (https://phabricator.wikimedia.org/T189475) (owner: 10Amire80) [13:02:13] mforns' first comment is easy. [13:02:32] the second is about a line that I just copied from another config file, which milimetric had written :) [13:02:44] Maybe it can be simply removed. [13:06:47] (03CR) 10Amire80: Add a scheduled job for daily CX abuse filters statistics (031 comment) [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/475618 (https://phabricator.wikimedia.org/T189475) (owner: 10Amire80) [13:08:05] (03PS2) 10Amire80: Add a scheduled job for daily CX abuse filters statistics [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/475618 (https://phabricator.wikimedia.org/T189475) [13:30:17] hi aharoni! thanks for the fixes [13:32:27] I think having a lag specified (second comment) was a good idea, but 85400 seconds (1 day) was probably too much. I would use 1 hour, so -> lag: 3600 # wait 1 hour to compute last day [13:34:07] mforns: ack, I'll update the patch [13:34:24] aharoni, thanks, I also added the comment in the patch [13:34:40] (03CR) 10Mforns: Add a scheduled job for daily CX abuse filters statistics (031 comment) [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/475618 (https://phabricator.wikimedia.org/T189475) (owner: 10Amire80) [13:35:28] aharoni, I think we are able now to test this. Do you want to pair via Hangouts some time? [13:39:10] Yes, in the next few minutes. Let me just update the patch. [13:39:29] (Oh, and I have to reconnect to IRC, wait just a minute.) [13:40:35] back [13:47:35] um [13:47:37] weird [13:47:41] reconnecting again [14:03:13] (03PS3) 10Amire80: Add a scheduled job for daily CX abuse filters statistics [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/475618 (https://phabricator.wikimedia.org/T189475) [14:05:01] mforns: I updated the patch. Hangout whenever you're ready [14:34:04] mforns: beep [14:48:55] aharoni, sorry I was having quick lunch, now back! [14:49:22] do you want to meet now? [15:01:10] mforns: now is good [15:01:21] aharoni, great [15:02:01] aharoni, To join the video meeting, click this link: https://meet.google.com/wey-upgj-qeg [15:02:01] Otherwise, to join by phone, dial +1 914-359-6313 and enter this PIN: 450 962 864# [15:02:24] uau, sorry for the noise, I thought it would paste the url only [15:10:04] (03PS1) 10Fdans: Expose offset and underestimate numbers in uniques [analytics/aqs] - 10https://gerrit.wikimedia.org/r/476033 (https://phabricator.wikimedia.org/T164201) [15:11:44] ottomata: o/ sorry i missed yer ping yesterday [15:13:07] phuedx: hiya np! [15:13:12] shall we look now? [15:13:34] ottomata: suresies! as long as it's alright with you [15:14:43] ottomata: actually, gimme 5-10 to get coffee and a snack [15:14:51] ya for sure! [15:14:59] ok phuedx i'm going to run home real quick then too, be back in about the same... [15:18:10] (03PS4) 10Amire80: Add a scheduled job for daily CX abuse filters statistics [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/475618 (https://phabricator.wikimedia.org/T189475) [15:20:50] ok here whenever [15:42:59] joal, yt? [15:43:44] ottomata: hey, sorry [15:43:49] hiya [15:44:16] so what are we looking for phuedx? [15:44:21] my eldest son is ill and he just woken up :) [15:44:24] (only been half paying attentiont to that ticket) [15:44:27] oh ok! no worries! [15:44:33] ottomata: might be easier to jump in a hangout [15:44:33] i'm just starting my day so i'll be on for a while [15:44:43] we have a standup starting in 15 tho. [15:44:45] sure, now is ok then? [15:44:57] nah! he's laying on a sofa colouring in at the moment [15:44:59] sure [15:45:02] ok cool [15:45:05] come on into the batcave!~ [15:45:06] https://hangouts.google.com/hangouts/_/wikimedia.org/a-batcave [15:55:40] Hey ottomata - let's talk about the refinement-schema issue post standup if ok? [15:55:45] yuppers [15:55:55] joal: catch up on T209178 pre-standup? [15:55:55] T209178: Refactor Mediawiki-Database ingestion - https://phabricator.wikimedia.org/T209178 [15:56:01] i think i have a working fix, but ihave some qs about what to do in some other array cases [15:56:09] yes milimetric ! OMW [15:56:42] bc-2 milimetric ? [16:01:14] ping joal milimetric mforns [16:01:18] standddupppp [16:01:21] sorry coming! [16:02:04] phuedx: heyyyaa https://logstash.wikimedia.org/app/kibana#/dashboard/default?_g=h@7bf0c26&_a=h@ecf0ee1 [16:02:06] i think its working? [16:02:11] ottomata: i see it working! [16:02:22] sweet! [16:03:01] thanks :) [16:03:25] ottomata: any recommendations for a "logstash person"? [16:05:05] hahhhh hmmmm [16:05:13] maybe godog (filippo)? [16:05:17] or at least he'd know who to ask i think [16:13:07] phuedx: godog and herron [16:13:18] moritzm: ty! [16:13:38] yw :-) [16:38:55] mforns: when you're ready :) [16:39:13] aharoni, in round 20 mins [16:39:15] is that ok [16:39:16] ? [16:39:47] yes [16:54:25] (03PS2) 10Michael Große: Update metric's items and properties automatically [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/475807 (https://phabricator.wikimedia.org/T209399) [16:58:43] (03PS3) 10Michael Große: Update metric's items and properties automatically [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/475807 (https://phabricator.wikimedia.org/T209399) [17:00:24] (03CR) 10Michael Große: "This change is ready for review." [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/475807 (https://phabricator.wikimedia.org/T209399) (owner: 10Michael Große) [17:02:40] aharoni, just 3 more minutes [17:04:44] I'm here [17:05:27] aharoni, ok, me too [17:05:40] should we use the same hangouts link as before? [17:05:45] let me try... [17:06:21] aharoni, https://meet.google.com/wey-upgj-qeg [17:07:07] folks logging off to run a bit, my brain is fried after today's tests :D [17:07:15] * elukey off! [17:07:26] (will read later if anybody needs me!) [17:19:31] HaeB which analyst email list should I use? [17:19:43] produce-analytics or data-analysts ? [17:19:55] product-analytics* [17:20:21] ah found your email and answered myself, sorry for ping! [17:20:32] mforns: looks like you disconnected [17:24:02] 10Analytics, 10Analytics-Wikimetrics, 10Patch-For-Review, 10WorkType-Maintenance: flake8 errors on wikimetrics - https://phabricator.wikimedia.org/T210320 (10Nuria) We have noted this on teh patches but this project is on life-support so you might not get timely responses to these patches or questions. Ple... [17:26:59] 10Analytics, 10Analytics-EventLogging, 10Patch-For-Review: Resurrect eventlogging_EventError logging to in logstash - https://phabricator.wikimedia.org/T205437 (10Ottomata) https://logstash.wikimedia.org/goto/bda91f37481ae4970ee21e11810d49d3 [17:27:09] 10Analytics, 10Analytics-Wikimetrics, 10Patch-For-Review, 10WorkType-Maintenance: flake8 errors on wikimetrics - https://phabricator.wikimedia.org/T210320 (10rafidaslam) @Nuria no problem. This isn't a big issue anyway, this can be fixed very easy anytime. We only need the answer to this question: ` When I... [17:27:30] 10Analytics, 10Analytics-EventLogging, 10Patch-For-Review: Resurrect eventlogging_EventError logging to in logstash - https://phabricator.wikimedia.org/T205437 (10phuedx) >>! In T205437#4778368, @Ottomata wrote: > https://logstash.wikimedia.org/goto/bda91f37481ae4970ee21e11810d49d3 https://logstash.wikimedi... [17:27:34] 10Analytics, 10Analytics-EventLogging, 10Patch-For-Review: Resurrect eventlogging_EventError logging to in logstash - https://phabricator.wikimedia.org/T205437 (10phuedx) 05Open>03Resolved a:03phuedx Great to see this working! Thanks for all of your help @Ottomata and @fgiunchedi. [17:30:52] 10Analytics, 10Analytics-Kanban: Refactor Sqoop, join actor and comment from analytics replicas - https://phabricator.wikimedia.org/T210522 (10Milimetric) p:05Triage>03High [17:59:21] (03PS5) 10Mforns: Add a scheduled job for daily CX abuse filters statistics [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/475618 (https://phabricator.wikimedia.org/T189475) (owner: 10Amire80) [18:00:38] (03PS6) 10Mforns: Add a scheduled job for daily CX abuse filters statistics [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/475618 (https://phabricator.wikimedia.org/T189475) (owner: 10Amire80) [18:04:50] 10Analytics, 10EventBus, 10Operations, 10WMF-JobQueue, and 6 others: Kafka eqiad.mediawiki.page-delete topic is empty - https://phabricator.wikimedia.org/T210451 (10Smalyshev) Yep, seeing the events in grafana now, so I think it's all good now. Thanks! [18:06:31] elukey, ottomata: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/475984/ can probably also be merged without SRE meeting approval, it's just some fine-tuning of existing permissions, if you need it ealier than next Monday, simply ping Mark or Faidon for a quick sign-off and merge away [18:06:31] 10Analytics, 10EventBus, 10Operations, 10WMF-JobQueue, and 6 others: Kafka eqiad.mediawiki.page-delete topic is empty - https://phabricator.wikimedia.org/T210451 (10Pchelolo) Do you need the events for the last month to be replayed? [18:08:02] (03PS7) 10Mforns: Add a scheduled job for daily CX abuse filters statistics [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/475618 (https://phabricator.wikimedia.org/T189475) (owner: 10Amire80) [18:10:35] (03CR) 10Mforns: [C: 031] "LGTM! And is tested. Please, feel free to merge." [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/475618 (https://phabricator.wikimedia.org/T189475) (owner: 10Amire80) [18:14:16] 10Analytics, 10EventBus, 10Operations, 10WMF-JobQueue, and 6 others: Kafka eqiad.mediawiki.page-delete topic is empty - https://phabricator.wikimedia.org/T210451 (10Smalyshev) @Pchelolo No I already updated the affected items manually. [18:18:16] (03CR) 10Amire80: [C: 032] Add a scheduled job for daily CX abuse filters statistics [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/475618 (https://phabricator.wikimedia.org/T189475) (owner: 10Amire80) [18:18:22] (03Merged) 10jenkins-bot: Add a scheduled job for daily CX abuse filters statistics [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/475618 (https://phabricator.wikimedia.org/T189475) (owner: 10Amire80) [18:26:32] hey, does anyone know how do i get access to pikwik? i have wikitech login but it doesn't work. it works for turnilo [18:26:38] cc @elukey [18:27:31] nzr_: i did just find https://wikitech.wikimedia.org/wiki/Analytics/Systems/Piwik#Access [18:27:41] not sure if that is helpful, i assume you are in those groups [18:27:46] ? [18:27:50] maybe mfournier do you know more? [18:27:52] oops [18:27:54] sorry wrong ping [18:28:01] nuria: do you know? [18:40:44] nzr_: what's your wikitech/ldap username? [18:40:50] nirzar [18:40:55] ottomata: [18:41:27] ottomata: IIRC we had to create users on piwik itself no? [18:43:30] we definitely need some docs at https://wikitech.wikimedia.org/wiki/Analytics/Systems/Piwik#Access [18:44:11] nzr_: are you getting denied at the http auth level? [18:44:44] elukey: just verified, nzr is in the wmf ldap group [18:44:48] https://usercontent.irccloud-cdn.com/file/otYzkuRd/image.png [18:45:02] yeah but that gets him passing the LDAP auth, not the piwik login [18:45:30] right nzr_ ? You are able to login with ldap when the user/pass menu pops out, but then not in piwik itself [18:45:30] ah ok, so ya he needs an account created, i think I do too (or do I?) [18:45:39] elukey, can you do a quick review of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/476081/ ? It's super simple [18:45:39] ya elukey that's right based on screen shot he just sent [18:45:52] > right nzr_ ? You are able to login with ldap when the user/pass menu pops out, but then not in piwik itself [18:45:52] neither [18:45:56] mforns_: i can do that [18:45:58] ottomata: there's a piwik-wmf-admin in pwstore, I am now in the admin panel of piwik (sooo slow) [18:46:00] not even on the pop yp [18:46:09] k! [18:46:13] nzr: there is a piwik user taht was created for design [18:46:21] cc elukey [18:46:21] ah! [18:46:38] thanks otto!! [18:47:07] 10Quarry, 10Patch-For-Review: Quarry should refuse to save results that are way too large - https://phabricator.wikimedia.org/T188564 (10zhuyifei1999) p:05High>03Unbreak! This is getting [[https://tools.wmflabs.org/nagf/?project=quarry|ridiculously bad]] with queries like https://quarry.wmflabs.org/query/... [18:47:16] nzr_: what site are you trying to look at? [18:47:48] nuria: where are the pass for the users stored? [18:47:58] elukey: in piwik itself [18:48:06] wikimediafoundation.org [18:48:11] sure sure, I mean how can I retrieve them : [18:48:13] :) [18:49:02] the admin panel is horribly slow for me [18:49:12] elukey: ya, it always is [18:49:40] fyi: my same login works for turnilo [18:50:46] nzr_: right, cause every web that we host will require ldap [18:53:24] nuria: try now the admin panel [18:53:28] should be way quicker [18:53:31] ya, better [18:53:39] (03PS2) 10Milimetric: [WIP] working on understanding and testing page history and quality [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/468678 [18:53:48] Error in Matomo: curl_exec: Failed to connect to plugins.matomo.org port 443: Connection timed out. Hostname requested was: plugins.matomo.org [18:54:02] that was 10s wait before proceeding with the rest [18:54:20] elukey: what did you do? [18:54:31] https://matomo.org/faq/troubleshooting/faq_16646/ [18:54:35] need to puppetize it now [18:55:13] 10Quarry, 10Patch-For-Review: Quarry should refuse to save results that are way too large - https://phabricator.wikimedia.org/T188564 (10zhuyifei1999) quarry-worker-02 was practically dead. [18:55:20] mmm still very slow [18:55:30] elukey: enable_internet_features = 0? [18:55:35] I need to tune it a bit more tomorrow [18:55:37] yeah! [18:55:37] elukey: it is OK now really [18:58:14] nuria: really sorry I didn't notice it before :( [18:58:30] ottomata: Heya - Back from diner [18:58:34] elukey: i did notice it and DID NOT SEARCHED for thsi solution [18:58:36] *this [18:58:42] elukey: totally my bad [18:58:46] (03CR) 10jerkins-bot: [V: 04-1] [WIP] working on understanding and testing page history and quality [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/468678 (owner: 10Milimetric) [19:00:21] elukey: documented now: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Piwik#Access [19:00:40] woundeful thanks :) [19:01:50] I also disable the marketplace that seemed not needed [19:04:10] * elukey dinner :) [19:18:08] joal heya just started trying a bit ago [19:18:11] looking good [19:18:20] i think i can even 'merge' primitive map keys... maybe! [19:18:26] ottomata: I have added stuff in unit test, but the conversion seems to work ok [19:18:33] still trying though [19:18:35] awesome [19:33:59] ottomata: last test tells me that for structs in Maps, the struct must not change in term of structure, but primitive types can [19:34:48] Meaning if you use a struct as key or value, inner-primitive types can be evolved, but not the structure of the inner objects [19:36:26] (03PS1) 10Joal: Update the unit-test for Dataframe conversion [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/476093 (https://phabricator.wikimedia.org/T210465) [19:36:30] ottomata: --^ [19:36:42] oh interesting [19:37:05] hm [19:37:08] ottomata: My saying is not documented as unit-test, just the working cases [19:37:14] right [19:37:16] intresting [19:37:24] Do you want me to add a test? [19:37:25] that's a bit harder then, its another special case [19:37:29] right [19:38:07] ottomata: I actually realize I have not tested struct-change for arrays [19:38:22] doing now [19:38:41] failed ! [19:38:43] Similar issue [19:39:07] same thing as map values? [19:39:11] wait [19:39:15] same [19:39:27] oh hm but that's only if a cast happens, right? [19:39:30] cannot cast array> to array> [19:39:32] if we alter the table before hand via merge [19:39:36] yes, that makes sense [19:39:42] but, if we alter the table before [19:39:44] it won't have to case [19:39:48] because datatypes will be the same [19:39:56] shoudl be the same for map values then, ya? [19:41:00]