[01:02:13] (03PS1) 10Paul Kernfeld: Use pid library to manage Pidfile [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/623470 (https://phabricator.wikimedia.org/T173604) [01:10:04] 10Analytics, 10Patch-For-Review, 10good first task: Reportupdater: do not write execution control files in source directories - https://phabricator.wikimedia.org/T173604 (10paulkernfeld) a:03paulkernfeld I saw this was marked with "good first issue," so I put together a patch for the pidfile portion of thi... [06:18:17] 10Analytics: Ensure Puppet checks types as part of the build - https://phabricator.wikimedia.org/T261693 (10elukey) @razzi the puppet catalog compiler reports the error though: https://puppet-compiler.wmflabs.org/compiler1002/24823/an-worker1078.eqiad.wmnet/index.html The build fails (see title) and also if yo... [06:23:12] good morning [06:24:40] 10Analytics-Clusters, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-09-15) upgrade/replace memory in stat100[58] - https://phabricator.wikimedia.org/T260448 (10elukey) @Jclark-ctr I'd need to schedule the maintenance in advance to let people know that we are rebooting (a lot of users use these hosts)... [06:30:37] Hi team [06:32:04] 10Analytics-Clusters, 10Operations, 10ops-eqiad, 10User-Elukey: replace onboard NIC in kafka-jumbo100[1-6] - https://phabricator.wikimedia.org/T236327 (10elukey) @Jclark-ctr they will not but we can do one host at the time anyway when you have time! [06:41:09] 10Analytics, 10Patch-For-Review: Fix TLS certificate location and expire for Hadoop/Presto/etc.. and add alarms on TLS cert expiry - https://phabricator.wikimedia.org/T253957 (10elukey) >>! In T253957#6423681, @jbond wrote: >>>! In T253957#6351773, @elukey wrote: >> I recently discovered that we have `base::ex... [06:41:47] elukey: hello :) I have a virtual coffee to share (before 10am) if you wish :) [06:43:55] joal: sure! I am kicking off a reimage, maybe in 15? [06:44:05] when ou wish elukey :) [06:46:29] 10Analytics-Clusters: Upgrade Kafka Brokers to Debian Buster - https://phabricator.wikimedia.org/T255123 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts: ` ['kafka-jumbo1002.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/202009010646_el... [06:48:00] (03CR) 10Joal: "Missing the jar version in: spark_job_jar in bundle.properties, except from that looks good" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/623456 (https://phabricator.wikimedia.org/T257691) (owner: 10Nuria) [06:49:57] (03CR) 10Joal: [C: 03+1] "I didn't check the ordering but the number of lines is correct as well as the format - looks ok for me. Let's ask someone else in the team" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/623141 (https://phabricator.wikimedia.org/T253439) (owner: 10Gerrit maintenance bot) [06:58:05] (03CR) 10Joal: [C: 03+1] "Code looks good, one nit on commit message. And a question: EventLogging stuff still belongs in this repo as it is not 'the real' event th" (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/622369 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [07:00:04] 10Analytics-Radar, 10MediaWiki-extensions-NavigationTiming, 10Performance-Team (Radar): Invalid EventLogging messages for NavigationTiming topic - https://phabricator.wikimedia.org/T261665 (10awight) Sounds reasonableβ€”so the fix is to relax validation, correct? [07:07:07] 10Analytics-Clusters: Upgrade Kafka Brokers to Debian Buster - https://phabricator.wikimedia.org/T255123 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['kafka-jumbo1002.eqiad.wmnet'] ` Of which those **FAILED**: ` ['kafka-jumbo1002.eqiad.wmnet'] ` [07:19:21] 10Analytics-Clusters: Upgrade Kafka Brokers to Debian Buster - https://phabricator.wikimedia.org/T255123 (10elukey) Something interesting happens when wmf-auto-reimage runs puppet for the first time (via cumin) after the reimage, namely that the process seems stuck indefinitely. I have to use `install_console` o... [07:19:47] joal: ok I am ready [07:20:04] elukey: I have seen the message above :( [07:20:17] elukey: to the cave! [07:27:13] 10Analytics-Radar, 10MediaWiki-extensions-NavigationTiming, 10Performance-Team (Radar): Invalid EventLogging messages for NavigationTiming topic - https://phabricator.wikimedia.org/T261665 (10Gilles) No, because those invalid integers were causing processing issues further in the EventLogging pipeline, killi... [08:10:44] going for errand - back in ~1h30 [08:21:50] * elukey coffee [08:36:48] 10Analytics-Radar, 10Operations, 10Traffic, 10Patch-For-Review: Package varnish 6.0.x - https://phabricator.wikimedia.org/T261632 (10ema) Varnish 6.0.x fails to start in labs with our current setup: ` Sep 01 07:42:53 traffic-cache-atstext-buster varnish-frontend[7196]: rm: cannot remove '_.vsm_mgt/_.Arg.8c... [08:37:21] 10Analytics-Radar, 10MediaWiki-extensions-NavigationTiming, 10Performance-Team (Radar): Invalid EventLogging messages for NavigationTiming topic - https://phabricator.wikimedia.org/T261665 (10awight) >>! In T261665#6425937, @Gilles wrote: > invalid integers were causing processing issues further in the Event... [09:04:42] PROBLEM - Hue CherryPy python server on an-tool1009 is CRITICAL: PROCS CRITICAL: 0 processes with command name python2.7, args /usr/lib/hue/build/env/bin/hue runcherrypyserver https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hue/Administration [09:07:06] PROBLEM - Hue Kerberos keytab renewer on an-tool1009 is CRITICAL: PROCS CRITICAL: 0 processes with command name python2.7, args /usr/lib/hue/build/env/bin/hue kt_renewer https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hue/Administration [09:08:04] ah! this is me --^ [09:08:07] testing host [09:15:12] back [09:27:46] (03CR) 10Joal: "Comments inline - mostly variable-names and pom suggestions." (0310 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/623448 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [09:31:33] elukey: Should we unset camus-checker as long as the main DC is codfw (or make it use codfw topic)? [09:35:16] joal: yes I think it is a good idea, I wanted to wait for Andrew to understand what he wants to do [09:35:29] ack! [09:54:11] 10Analytics, 10Patch-For-Review: Fix TLS certificate location and expire for Hadoop/Presto/etc.. and add alarms on TLS cert expiry - https://phabricator.wikimedia.org/T253957 (10jbond) > Yes it may be enough, I'll test it on the Hadoop test cluster. IIUC in the patch you "package" the private key into a p12 fi... [11:52:23] * elukey afk for late lunch break + errand [12:00:10] (03PS1) 10Joal: Update drop-mediawiki-snapshots parameters and datasets [analytics/refinery] - 10https://gerrit.wikimedia.org/r/623586 (https://phabricator.wikimedia.org/T237047) [12:52:21] ottomata: yay! thanks! [13:15:20] 10Analytics-Radar, 10Operations, 10Traffic, 10Patch-For-Review: Package varnish 6.0.x - https://phabricator.wikimedia.org/T261632 (10Vgutierrez) [13:26:29] 10Analytics-Radar, 10MediaWiki-extensions-NavigationTiming, 10Performance-Team (Radar): Invalid EventLogging messages for NavigationTiming topic - https://phabricator.wikimedia.org/T261665 (10Ottomata) > This is a separate problem: the service should be more robust, logging warnings for bad input records but... [13:33:40] 10Analytics-Radar, 10MediaWiki-extensions-NavigationTiming, 10Performance-Team (Radar): Invalid EventLogging messages for NavigationTiming topic - https://phabricator.wikimedia.org/T261665 (10Gilles) I'd rather have all filtering of out of bounds value happen in the same place. Both huge and negative values... [13:39:15] (03PS2) 10Joal: Update drop-mediawiki-snapshots parameters and datasets [analytics/refinery] - 10https://gerrit.wikimedia.org/r/623586 (https://phabricator.wikimedia.org/T237047) [13:39:38] (03CR) 10Joal: [V: 03+1] "Tested in dry-run mode" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/623586 (https://phabricator.wikimedia.org/T237047) (owner: 10Joal) [13:43:38] switchover starting in a few minutes, no prod changes for the moment please :) [13:44:15] ack elukey! [13:46:02] (03PS5) 10Ottomata: Use EventSchemaLoader from org.wikimedia.eventutilities [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/622369 (https://phabricator.wikimedia.org/T251609) [13:46:47] (03CR) 10Ottomata: "> EventLogging stuff still belongs in this repo as it is not 'the real' event thing, but a legacy maintenance while not yet merged - corre" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/622369 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [13:49:19] (03CR) 10jerkins-bot: [V: 04-1] Use EventSchemaLoader from org.wikimedia.eventutilities [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/622369 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [14:00:39] 10Analytics, 10Analytics-Data-Quality: page_id is null where it shouldn't be in mediawiki history - https://phabricator.wikimedia.org/T259823 (10JAllemandou) I have investigated quickly and we should take page_artificial_id into consideration: most events having page-id `null` or `0` have a defined page-artifi... [14:01:42] 10Analytics: Productionize analysis of editcount vs per_user_revision_count - https://phabricator.wikimedia.org/T168648 (10JAllemandou) This task is still real. We have never taken the time to check the difference and investigate. Linking `mediawiki-history` and `event` where edit-counts from the DB is present w... [14:24:09] 10Analytics, 10Analytics-Kanban, 10Privacy Engineering, 10Product-Analytics, and 3 others: Drop data from Prefupdate schema that is older than 90 days - https://phabricator.wikimedia.org/T250049 (10Milimetric) Apologies this took so long. My plan is to apply the [[ https://gerrit.wikimedia.org/r/c/mediawi... [14:26:11] 10Analytics, 10Analytics-Data-Quality: page_id is null where it shouldn't be in mediawiki history - https://phabricator.wikimedia.org/T259823 (10Milimetric) I thought we excluded these from denormalized, and left them just in page history. I can see why we'd want to include them. Hm... it is very confusing.... [14:39:16] Reading along in -sre and -ops is like listening in on old apollo mission chatter from CAPCOM. Except, I understand very little :) [14:42:15] I think that only few people have all the pieces in mind (I am not one of them) [14:43:33] hopefully in the future the mediawiki app layer will be active in both dcs all the time :) [14:57:25] (03CR) 10Milimetric: [C: 04-1] "Just two small nits. Really nice change! I love how the code became simpler in the same way the configs can now become simpler. That's " (032 comments) [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/623060 (https://phabricator.wikimedia.org/T193171) (owner: 10Paul Kernfeld) [15:00:07] 10Analytics-Clusters: Establish what data must be backed up before the HDFS upgrade - https://phabricator.wikimedia.org/T260409 (10Milimetric) @JAllemandou, responding while Marcel is gone. I looked through and found a few of the reports rely on data that's changing (so if lost, would be hard/impossible to reco... [15:06:35] elukey: if this were true: https://phabricator.wikimedia.org/T233336#6414979, then how come it works when testing the URL shortener config change locally? [15:06:40] it's still executing on the server [15:06:44] (without logging in) [15:07:30] somehow the IP range of the server is on the block list for that wiki? [15:07:44] but the IP of the an-tool I tested on is not? [15:07:56] (sorry just thinking out loud) [15:11:26] milimetric: mmm interesting.. when you say "locally", what do you mean? (host, command, etc..) [15:12:38] aaah, you're right, I always got the error, I forgot I had it working with tinyurl [15:12:44] um... maybe we just use tinyurl? [15:12:54] no... not a good idea [15:13:01] oh... well, people are doing that anyway... [15:13:18] (03CR) 10Ottomata: "Will update this patch to also use EventLoggingSchemaLoader from event-utilities after https://gerrit.wikimedia.org/r/c/wikimedia-event-ut" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/622369 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [15:14:25] milimetric: I'd prefer our internal service if possible, but if too hard we can think to use tiny url [15:14:48] I mean... this shouldn't be blocked... that's the real problem [15:14:57] the risk would be to leak something via the URL using a thirdparty tool, but very rare use case probably... [15:14:57] right? [15:15:40] for internal traffic it shouldn't in theory [15:16:14] right, I agree, and people are doing it anyway (using bitly and so on), so hopefully we can figure out how to unblock this IP, would make everything more secure [15:16:18] elukey: is the IP in the list I linked [15:18:21] milimetric: elukey: something I did want to ask re: Turnilo: how is data encoded in the URL? it's not unusual that (when diagnosing DoS or such) we pass around links that filter based on certain src IPs [15:18:54] cdanis: yeah good point, these use cases were the ones that I am worried about [15:19:07] no idea, never checked [15:19:10] RhinosF1: nope [15:19:41] elukey: then that makes zero sense [15:19:46] cdanis: it's not safe in any way, so I wouldn't pass it to any third party service [15:20:00] (I don't know exactly but it's like base 64 encoded or something similar) [15:21:49] cdanis: do you know who we'd talk to about checking why this server is blocked? [15:22:41] milimetric: if it's on a mediawiki block list... hm. I am going to redirect you to Reedy because he'll at least know where to look and who to ask πŸ™ƒ [15:23:42] is there a TLDR? :P [15:26:43] 10Analytics-Radar, 10Better Use Of Data, 10Product-Analytics, 10Product-Infrastructure-Data, and 3 others: Session Length Metric. Web implementation - https://phabricator.wikimedia.org/T248987 (10jlinehan) [15:27:19] Reedy: we are trying to use the meta's URL shortner for turnilo's URLs, and wondering if it needs authentication or not (or if there is some list of approved internal ips that can create short urls without it etc..) [15:27:47] klausman: in ~1h your credentials should be everywhere :) [15:28:02] https://noc.wikimedia.org/conf/highlight.php?file=CommonSettings.php and $wgUrlShortenerApprovedDomains [15:28:23] and $wgUrlShortenerAllowedDomains [15:28:52] (03CR) 10Ottomata: Add ProduceCanaryEvents job (039 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/623448 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [15:28:58] (03PS3) 10Ottomata: Add ProduceCanaryEvents job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/623448 (https://phabricator.wikimedia.org/T251609) [15:28:59] klausman: I am running puppet on stat1004, when you have time try to ssh to it [15:29:21] (stat1004.eqiad.wmnet) [15:30:25] Reedy: yep turnilo.wikimedia.org should be ok, but when we try it from one analytics node we get "Blocked users can't make short URLs." [15:30:43] more info in https://phabricator.wikimedia.org/T233336 [15:31:00] I bet it's some block to stop cluster IPs "editing" [15:31:10] If it's a MW error, that is [15:32:11] RhinosF1 pointed out these configs https://phabricator.wikimedia.org/T233336#6414976 but I have zero knowledge of mw [15:32:15] (03CR) 10jerkins-bot: [V: 04-1] Add ProduceCanaryEvents job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/623448 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [15:32:31] joal: so, parent pom deps aren't automatically inheritied by the child modules? [15:34:12] Reedy, yeah it seems like that. I can curl the shortner like this and it works: [15:34:13] curl 'https://meta.wikimedia.org/w/api.php?action=shortenurl&format=json' -X POST --data [15:34:13] "url=http://turnilo.wikimedia.org/#wiki/4/N4IgbglgzgrghgGwgLzgFwgewHYgFwhqZqJQgA0408SqGOAygKZobYDmZe2MCClGALZNkOJvhABRNAGMA9AFUAKgGEKIAGYQEaJgCcuAbVBoAngAdxBIeMp6mGiTfU2ACvqwATI6E8w96Fi4BK4AjAAi6lC65vgAtKECFlYgCOhM0SAAvgC6uZRQ5khoRjkFHhn4xiD2GvpM2DIpcJ6eTJ7qGph6guj4JskSbRpwvGjqYIgwKdlJlhLCcLD22WUg5hDY2O3hEMLYUEFGIDIAFnBbTAjq2HDChXBN6tAASpgARsQgaxuXngzdcYEFptDpZIA=" [15:34:39] There's quite a bit of abstraction in the block code [15:35:02] But seems quite likely it's something along those lines [15:35:13] but if I basically do the same thing, using the api-rw.discovery.wmnet proxy, as in https://gerrit.wikimedia.org/r/c/operations/puppet/+/622600/2/modules/turnilo/templates/config.yaml.erb, it gives that error [15:35:45] (sorry, using http://webproxy.eqiad.wmnet:8080 as the proxy) [15:36:33] (03PS7) 10Paul Kernfeld: reader.py: Get all report keys from defaults [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/623060 (https://phabricator.wikimedia.org/T193171) [15:39:22] elukey: doesn't this match the install1003.eqiad.wmnet address? https://github.com/wikimedia/operations-mediawiki-config/blob/e0969147a87cdb961c07abd39d7dd66554e9b58c/wmf-config/reverse-proxy.php#L30 [15:40:45] yep, it is [15:41:11] ok, so that's the problem, it's blocked explicitly as RhinosF1 said [15:42:05] (03CR) 10Paul Kernfeld: "> Patch Set 6: Code-Review-1" (031 comment) [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/623060 (https://phabricator.wikimedia.org/T193171) (owner: 10Paul Kernfeld) [15:43:26] milimetric: that is a codfw subnet, i cannot match an eqiad ip no? [15:44:13] (03CR) 10Milimetric: [V: 03+2 C: 03+2] "Oh! I missed that actually, but I like your new phrasing better. Thanks!" [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/623060 (https://phabricator.wikimedia.org/T193171) (owner: 10Paul Kernfeld) [15:44:39] ip for install1002 [15:44:44] err 1003 is 208.80.154.32 [15:44:51] https://github.com/wikimedia/operations-mediawiki-config/blob/e0969147a87cdb961c07abd39d7dd66554e9b58c/wmf-config/reverse-proxy.php#L11 [15:45:07] oh, I get 2620:0:861:1:208:80:154:32 [15:45:13] oh! That's because of the switch right now! [15:45:14] hahaha [15:45:19] uh... that makes everything confusing [15:46:53] milimetric: maybe we should just try to contact the apis via the external route, so meta.wikimedia.org [15:47:25] I thought we couldn't, that we had to go through the proxy? [15:50:07] yes exactly, we should use the eqiad proxy [15:57:58] SSH functional confirmed [16:02:09] ping klausman : hello and standup! [16:03:44] omw [16:23:37] (03CR) 10Fdans: "Thanks for the lil changes :)" (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/612574 (https://phabricator.wikimedia.org/T255757) (owner: 10Fdans) [16:34:01] https://timeformachine.com/products/marvel-tank-2-metal-model-kit The kit I mentioned [17:09:22] ottomata: did you have a chance to look at my schema issues? [17:15:56] ah not yet cdanis will do today though! [17:16:27] ottomata: for the CR, are there anychnages i need to do on local pom.xml to use the version of eventutilities that i have cloned? [17:16:55] ottomata: ok! just fyi I'm on vacation ~all of next week so I'd love to get this all checked in this week, if possible [17:16:59] elukey: so the IP is in that list? What are you using to run your command? Could you perhaps authenticate? [17:17:24] (and then in 3rd week of Sept we can start turning on NEL for some wikis, I'm going to write up a plan but it will roughly follow deployment groups) [17:18:33] RhinosF1: nono it is the IP of the proxy, not the VM.. my idea is to go through the proxy and try to contact meta directly, let's see how it goes [17:19:02] elukey: I see. [17:19:11] milimetric: do you have a min to discuss the proxy thing? [17:19:21] sure! cave? [17:19:21] RhinosF1: thanks for the inputs :) [17:19:24] yep! [17:20:06] elukey: no problem, ping if you need more help [17:22:20] (03CR) 10Joal: "One minimal nit, and pom updates as discussed at standup" (034 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/623448 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [17:24:16] 10Analytics-Clusters: Establish what data must be backed up before the HDFS upgrade - https://phabricator.wikimedia.org/T260409 (10JAllemandou) > Why don't we just back up everything, the RU output is teeny tiny. Happy with that solution - Let's start documenting sizes in the task body. [17:27:29] Reedy: do you think we could simulate whatever "trusted header" refers to here? https://github.com/wikimedia/operations-mediawiki-config/blob/167a7bd1a1c7c5d2051c229e50d7525f1b745993/wmf-config/CommonSettings.php#L3730 [17:33:40] I see, https://meta.wikimedia.org/wiki/XFF_project [17:43:30] logging off for today! o/ [17:45:42] 10Analytics, 10Patch-For-Review: Add urlshortener button to Turnilo - https://phabricator.wikimedia.org/T233336 (10Milimetric) Ok, thanks @RhinosF1 for the help, here's the problem as I understand it: * Internal IPs [[ https://github.com/wikimedia/operations-mediawiki-config/blob/167a7bd1a1c7c5d2051c229e50d75... [17:51:13] milimetric: XFF headers are used by proxies to show the origin IP [17:51:49] right [17:52:05] but it looks like if the origin IP was an internal IP, it would still be blocked, right? [17:53:59] milimetric: yes if the origin of the request was also on the list it would still be blocked [17:58:12] https://blog.koehntopp.info/2020/08/31/on-touching-candles.html A great article a friend of mine wrote on how some skills need to be learned the hard way [17:59:01] right, yeah, that's helpful RhinosF1, and now I think we have a good handle on the problem, though we're out of ideas for how to solve. It seems like anything we do would be hacky, but that the block system is having unexpected side effects too [17:59:34] milimetric: WMF internal IPs should be trusted anyway for XFF fwiw, you just need to provide an IP in the request header not on the list (like your own personal) [17:59:56] The other option would be to build a script that auths to an account [18:00:28] hm, seems hacky to do, and this runs every time someone clicks a "shorten url" button. [18:00:40] the internal IPs aren't trusted if 10.0.0.0/8 is blocked, right? [18:01:25] milimetric: if it's in CDNServersNoPurge, it should be trusted for XFF headers [18:01:41] What code is actually ran when you click shorten URL [18:02:26] RhinosF1: it's basically running this JS: https://gerrit.wikimedia.org/r/c/operations/puppet/+/622600/2/modules/turnilo/templates/config.yaml.erb [18:04:08] milimetric: Either use the JavaScript to provide the personal IP of the user logged in as an XFF header or if it could support OAUTH add an auth header [18:49:01] the XFF header would hopefully be straightforward [19:15:17] 10Analytics, 10Product-Analytics, 10Research: Annotate pageview data to alert users that previously included mobile app pageview data is NOT included in refined pageview datasets - https://phabricator.wikimedia.org/T258535 (10SNowick_WMF) [19:30:27] 10Analytics, 10Product-Analytics, 10Research: Annotate pageview data to alert users that previously included mobile app pageview data is NOT included in refined pageview datasets - https://phabricator.wikimedia.org/T258535 (10SNowick_WMF) Affected Builds: Android Build 2.7.50318-r-2020-05-18 iOS Build 6.6.1... [19:31:14] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Epic: API pageview counts for 'Mobile app' are incorrect since switch to mobile-html - https://phabricator.wikimedia.org/T256508 (10SNowick_WMF) [19:31:29] 10Analytics-Radar, 10Product-Analytics (Kanban): Identify next steps for dealing with missing mobile app pageview counts - https://phabricator.wikimedia.org/T256804 (10SNowick_WMF) 05Openβ†’03Resolved Affected Builds: Android Build 2.7.50318-r-2020-05-18 iOS Build 6.6.1 1743 2020-06-05 Resolved Builds: Andr... [20:14:50] 10Analytics, 10MediaWiki-API, 10MediaWiki-Special-pages, 10Platform Team Workboards (Clinic Duty Team): Number of active users from ApiQueryAllUsers grossly different from Special:Statistics page - https://phabricator.wikimedia.org/T261290 (10holger.knust) [20:16:22] 10Analytics, 10MediaWiki-API, 10MediaWiki-Special-pages, 10Platform Team Workboards (Clinic Duty Team): Number of active users from ApiQueryAllUsers grossly different from Special:Statistics page - https://phabricator.wikimedia.org/T261290 (10holger.knust) p:05Triageβ†’03Medium [21:07:34] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10Platform Team Workboards (Initiatives): reportupdater Pingback reports are broken and need to be refactored - https://phabricator.wikimedia.org/T246154 (10CCicalese_WMF) The graphs look great! Thank you so much!! [21:14:18] 10Analytics, 10Analytics-EventLogging: Update pingback reports to use heartbeat pings to filter data - https://phabricator.wikimedia.org/T236178 (10CCicalese_WMF) 05Openβ†’03Resolved a:03CCicalese_WMF Thank you, @mforns! [21:30:09] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10Platform Team Workboards (Initiatives): reportupdater Pingback reports are broken and need to be refactored - https://phabricator.wikimedia.org/T246154 (10Nuria) 05Openβ†’03Resolved [21:30:30] 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services): Set up Jenkins maven release job for wikimedia-event-utilities like analytics/refinery/source - https://phabricator.wikimedia.org/T259898 (10Nuria) 05Openβ†’03Resolved [21:30:33] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events - https://phabricator.wikimedia.org/T251609 (10Nuria) [21:31:06] 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services): Set up Jenkins maven release job for wikimedia-event-utilities like analytics/refinery/source - https://phabricator.wikimedia.org/T259898 (10Nuria) No code patch but thi... [21:31:23] 10Analytics, 10Analytics-Kanban: Move the Analytics infrastructure to Debian Buster - https://phabricator.wikimedia.org/T234629 (10Nuria) [21:31:25] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade schema[12]00[12] to Debian Buster - https://phabricator.wikimedia.org/T255026 (10Nuria) 05Openβ†’03Resolved [21:31:48] 10Analytics, 10Analytics-Kanban: Tune up thresholds of data quality hourly alarms - https://phabricator.wikimedia.org/T251814 (10Nuria) 05Openβ†’03Resolved [21:31:50] 10Analytics, 10Analytics-Kanban: Add hourly resolution to data quality outage/censhorship alarms - https://phabricator.wikimedia.org/T249759 (10Nuria) [21:32:10] 10Analytics, 10Analytics-Kanban: Whitelist new VisualEditorFeatureUse fields - https://phabricator.wikimedia.org/T256048 (10Nuria) 05Openβ†’03Resolved [21:37:57] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Order mediawiki_history dumps by event_timestamp - https://phabricator.wikimedia.org/T254233 (10Nuria) 05Openβ†’03Resolved [21:38:25] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: [Spike] Spark job for digests-only mediawiki-history-reduced - https://phabricator.wikimedia.org/T212928 (10Nuria) 05Openβ†’03Resolved [21:38:27] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Define reduce calculations needed to compute active editors per project family - https://phabricator.wikimedia.org/T249751 (10Nuria) [21:38:30] 10Analytics, 10Analytics-Wikistats: Active Editors metric per project family - https://phabricator.wikimedia.org/T188265 (10Nuria) [21:38:32] 10Analytics, 10Analytics-Wikistats, 10Patch-For-Review: Create report for "articles with most contributors" in Wikistats2 - https://phabricator.wikimedia.org/T204965 (10Nuria) [21:44:39] (03PS6) 10Nuria: Use EventSchemaLoader from org.wikimedia.eventutilities [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/622369 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [21:52:09] ottomata: added a test file https://gerrit.wikimedia.org/r/c/analytics/refinery/source/+/622369 to the refinery-job package [21:53:19] ottomata: so build would pass, we can also change test file path, it seemed to make sense to add test resource to the package where the test is [21:55:46] (03CR) 10Nuria: [C: 03+1] Use EventSchemaLoader from org.wikimedia.eventutilities [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/622369 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [21:56:21] (03CR) 10Nuria: [C: 03+1] "Virtual +2, looks good but leaving @otto the best time to merge" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/622369 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [22:25:10] 10Analytics, 10good first task: [reportupdater] Allow defaults for all config parameters - https://phabricator.wikimedia.org/T193171 (10Nuria) @paulkernfeld @Milimetric seems that this change need a companion chnage in https://wikitech.wikimedia.org/wiki/Analytics/Systems/Reportupdater#Config_file can @paulker... [22:25:40] 10Analytics, 10Analytics-Kanban, 10good first task: [reportupdater] Allow defaults for all config parameters - https://phabricator.wikimedia.org/T193171 (10Nuria) [22:48:51] ah nuria on that one, i'll be removing EventLoggingSchemaLoader too, after we make a release of event-utilities [22:48:54] joseph convinced me [22:48:56] to move it there [22:59:34] ottomata: meaning that the tester and all go to the event utilities pkg? [23:09:37] 10Analytics: Ensure Puppet checks types as part of the build - https://phabricator.wikimedia.org/T261693 (10Ottomata) Ya, PCC is good, but Razzi created this ticket hoping that an obvious incorrect type like this could be caught by the Jenkins tests that run with every patch, rather than having to check manually. [23:15:41] ya [23:41:50] 10Analytics, 10Analytics-Kanban, 10good first task: [reportupdater] Allow defaults for all config parameters - https://phabricator.wikimedia.org/T193171 (10paulkernfeld) @Nuria thanks for pointing that out! I just made the companion wiki edit.