[00:00:12] (03CR) 10Dzahn: [V: 032 C: 032] Update svn.wikimedia.org links to phabricator [analytics/wikistats] - 10https://gerrit.wikimedia.org/r/316289 (https://phabricator.wikimedia.org/T64570) (owner: 10Paladox) [00:00:12] (03Merged) 10jenkins-bot: Update svn.wikimedia.org links to phabricator [analytics/wikistats] - 10https://gerrit.wikimedia.org/r/316289 (https://phabricator.wikimedia.org/T64570) (owner: 10Paladox) [00:00:20] (03CR) 10jenkins-bot: Update svn.wikimedia.org links to phabricator [analytics/wikistats] - 10https://gerrit.wikimedia.org/r/316289 (https://phabricator.wikimedia.org/T64570) (owner: 10Paladox) [00:01:41] 10Analytics, 10Analytics-Wikistats, 10Easy, 10I18n, 10Patch-For-Review: WikiReportsLocalizations.pm still fetches language names from SVN - https://phabricator.wikimedia.org/T64570#670885 (10Dzahn) Can you check on this again ? I just merged the change above. Replacing these links in other places was unc... [08:03:21] 10Analytics-Kanban, 10Operations, 10hardware-requests: Decommission stat1002.eqiad.wmnet - https://phabricator.wikimedia.org/T173097#3609800 (10elukey) [08:13:06] 10Analytics-Kanban, 10Operations, 10ops-eqiad: Decommission stat1003.eqiad.wmnet - https://phabricator.wikimedia.org/T175150#3609809 (10elukey) [08:27:20] 10Analytics, 10Operations, 10ops-eqiad: Remove stat1002 - https://phabricator.wikimedia.org/T173094#3609827 (10elukey) [08:27:23] 10Analytics-Kanban, 10Operations, 10hardware-requests, 10Patch-For-Review: Decommission stat1002.eqiad.wmnet - https://phabricator.wikimedia.org/T173097#3609829 (10elukey) [08:27:38] 10Analytics, 10Operations, 10ops-eqiad: Remove stat1002 - https://phabricator.wikimedia.org/T173094#3518439 (10elukey) [08:27:40] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Replacement of stat1002 and stat1003 - https://phabricator.wikimedia.org/T152712#3609832 (10elukey) [08:27:44] 10Analytics-Kanban, 10Operations, 10hardware-requests, 10Patch-For-Review: Decommission stat1002.eqiad.wmnet - https://phabricator.wikimedia.org/T173097#3518505 (10elukey) 05duplicate>03Open [08:28:37] 10Analytics, 10Operations, 10ops-eqiad: Remove stat1002 - https://phabricator.wikimedia.org/T173094#3518439 (10elukey) 05Open>03Resolved Mentioned in https://phabricator.wikimedia.org/T173097 [08:28:53] 10Analytics-Kanban, 10Operations, 10hardware-requests, 10Patch-For-Review: Decommission stat1002.eqiad.wmnet - https://phabricator.wikimedia.org/T173097#3518505 (10elukey) [08:29:52] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Replacement of stat1002 and stat1003 - https://phabricator.wikimedia.org/T152712#3609849 (10elukey) Puppet/Salt cleaned from stat1002/3, the last steps are for DC-ops in the related tasks. [08:51:54] 10Analytics-Kanban, 10Operations, 10hardware-requests, 10Patch-For-Review: Decommission stat1002.eqiad.wmnet - https://phabricator.wikimedia.org/T173097#3609884 (10elukey) I removed puppet/salt credentials and wiped puppet, but didn't proceed any further since I didn't want to mess with DC-Ops procedure. I... [08:52:09] 10Analytics-Kanban, 10Operations, 10ops-eqiad: Decommission stat1003.eqiad.wmnet - https://phabricator.wikimedia.org/T175150#3609885 (10elukey) I removed puppet/salt credentials, powered down and wiped puppet, but didn't proceed any further since I didn't want to mess with DC-Ops procedure. I was under the i... [09:31:19] * elukey errands + early lunch, reachable via phone! [09:32:30] * joal imagines elukey whistling 'call me maybe' [09:36:26] * elukey plays the song in his head while doing errands [09:47:48] https://www.youtube.com/watch?v=5meWI3iX1sE / https://www.youtube.com/watch?v=q1D6qWO-JzQ [11:07:51] :D [11:43:46] 10Analytics-Kanban, 10Analytics-Wikistats: Handle long project names in Wikiselector - https://phabricator.wikimedia.org/T173373#3610215 (10mforns) [11:50:58] mforns: o/ [11:51:01] https://gerrit.wikimedia.org/r/#/c/378236 :) [12:05:41] then we could also add another option like --daemon, so it will just run everything in a loop [12:06:39] eventlogging_sync has a similar approach, and its init.d script just writes a file in /var/run/etc.. to avoid multiple daemons spawned at the same time [13:20:04] phew, ok, it looks like I can work today [13:21:48] joal: I'm catching up with the change you merged (the mediawiki-history-reduced job) and looking at the other one after, any better way I can be useful to you? [13:21:59] fdans: I'll review your updates [13:22:00] milimetric: Hi ! [13:22:13] milimetric: Please get out anytime needed, right? [13:22:44] milimetric: +1 to joal :) [13:23:00] yeah, long story short though things are relatively under control [13:23:05] milimetric: That said, Would you mind reviewing the huge pack of doc I have been writing today to explain some choices/changes in mediawiki_history_reduced? [13:23:15] milimetric: Still some to do, but the text id done [13:23:18] joal: sure, I'll do that [13:23:29] where's it at? [13:23:55] milimetric: if you have a couple minutes to talk about bar graphs and their non-fixed width that would be great :) [13:24:13] fdans: let's go now [13:24:46] milimetric: omw [13:24:50] milimetric: when you have time - https://gist.github.com/jobar/0bc49d4005be123a56300e26fe38a354 [13:24:59] joal: will do, great [13:30:01] taking my break team- later [13:48:17] urandom: o/ [13:48:32] if you have a minute I'd need some help with the jmx exporter [13:48:53] I am trying to find out how to log its "logs" to understand why the config is not ok [13:48:56] :( [14:01:40] fdans, thanks for the review [14:01:42] :] [14:02:05] now, looking at the code again, I think I organized the methods a bit oddly [14:02:23] they should be within the 'methods' entry no? [14:05:45] mforns: yep, also, wouldn't $('input') select all the inputs in the document? [14:06:10] fdans, yes, you're right [14:06:30] for that I would put it inside methods and do $(this.$el, 'input') [14:07:21] yea, cool [14:10:55] 10Analytics-Kanban, 10Analytics-Wikistats: Use daily granularity for 1-month time ranges - https://phabricator.wikimedia.org/T173372#3525876 (10fdans) [14:12:06] fdans: do me a favor and pull back into in-progress anything you're still working on, so I know to review stuff if it's in the code review column [14:15:43] 10Analytics-Kanban, 10Analytics-Wikistats: Addition of Unique Devices metric - https://phabricator.wikimedia.org/T170461#3610508 (10fdans) [14:20:10] joal: reviewed the docs, have thoughts for you when you're free [14:20:34] (03CR) 10Nemo bis: Update svn.wikimedia.org links to phabricator (031 comment) [analytics/wikistats] - 10https://gerrit.wikimedia.org/r/316289 (https://phabricator.wikimedia.org/T64570) (owner: 10Paladox) [14:22:08] 10Analytics, 10Analytics-Wikistats, 10Easy, 10I18n, 10Patch-For-Review: WikiReportsLocalizations.pm still fetches language names from SVN - https://phabricator.wikimedia.org/T64570#3610549 (10Nemo_bis) a:03Paladox I cannot now run the script to verify, but just changing repository URL is ok as fix, exc... [14:25:12] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats2 bugs (3/4) - Data issues - https://phabricator.wikimedia.org/T170937#3610554 (10fdans) [14:25:50] milimetric: done [14:26:05] sweet, will review [14:29:33] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Audit users and account expiry dates for stat boxes - https://phabricator.wikimedia.org/T170878#3610577 (10elukey) Sent an email to all the remaining users to verify their need of the accounts :) [14:31:25] elukey: o/ [14:32:11] hi!!! I kinda figured out where the output goes, it was in syslog since kafka starts from a sh.. [14:32:35] I am currently trying to blacklist rates via patters :D [14:32:37] *patterns [14:32:54] elukey: rates? [14:33:12] a metric of somekind, i guess? [14:33:25] urandom: yes sorry metrics calculating rates directly (https://phabricator.wikimedia.org/P6007) [14:34:10] i see [14:34:58] I am using something like blacklistObjectNames: ["kafka_controller_controllerstats_?*rate:*", "..."] but I am failing :D [14:35:09] the syntax seems correct but no filtering [14:35:21] i haven't mucked with any of that yet [14:35:42] need to match some additional ones, actually [14:36:30] ahhh wait I might be stupid [14:36:37] I need to match the Mbeans names [14:36:45] I am currently matching the metrics [14:36:50] * elukey cries in a corner [14:36:54] oh, right [14:37:00] that makes sense :) [14:37:08] see urandom you appear and I find the error [14:37:14] haha [14:37:37] I am also really interested in moving AQS to the jmx exporter [14:38:53] elukey: yeah, i guess once we have restbase done, that should be pretty trivial [14:41:00] 10Analytics, 10Analytics-Wikistats: Provide yearly update of stats for audit report - https://phabricator.wikimedia.org/T174950#3610601 (10Erik_Zachte) 05Open>03Resolved Data have been relayed to Tony Sep 7. Thanks @Aklapper for quick response. [14:41:54] urandom: I also don't have a huge amount of time for the dashboard but I am happy to review/help if needed [14:44:29] elukey: i'd like to get outside my own echo chamber wrt what/how to visualize things, to be honest [14:44:57] elukey: so if you had some dashboards to use as examples, or some ideas there, that would go a long ways [14:45:18] otherwise, i imagine i'll end up recreating the dashboards we have, just for prometheus [14:45:38] (even if unintentionally :)) [14:46:03] we are pretty happy about https://grafana.wikimedia.org/dashboard/db/kafka [14:46:10] that shows up multiple kafka clusters [14:46:29] this is more or less what I meant with having a quickview of a cluster [14:46:38] to compare on the fly important metrics [14:46:45] then we can have super specific ones [14:46:51] 10Analytics, 10Analytics-Wikistats: Productionise list view - https://phabricator.wikimedia.org/T175265#3610604 (10fdans) [14:47:05] 10Analytics-Kanban, 10Analytics-Wikistats: Productionise list view - https://phabricator.wikimedia.org/T175265#3588157 (10fdans) [14:48:46] elukey: see, that's a sexy graph [14:48:54] s/graph/dashboard/ [14:57:59] fdans, I updated the animation code: https://phabricator.wikimedia.org/D785 [14:58:37] hey dudes, having a lot of home internet troubles [14:58:47] on the phone with internet people.. maybe be late to standup [14:59:22] Hmmm, vandalism destroyed a fiber cable! [14:59:23] yar [15:00:04] those vandals! [15:00:16] mforns: left just one comment [15:12:08] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Audit users and account expiry dates for stat boxes - https://phabricator.wikimedia.org/T170878#3610639 (10Etonkovidova) [15:26:27] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Audit users and account expiry dates for stat boxes - https://phabricator.wikimedia.org/T170878#3610685 (10egalvezwmf) Please keep access for me (chedasaurus). We use it for our quarterly metrics. Thanks! [15:34:56] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Audit users and account expiry dates for stat boxes - https://phabricator.wikimedia.org/T170878#3610700 (10KartikMistry) WMF staff. Backup of @Amire80 So, you can keep my access. Thanks! [15:55:46] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Audit users and account expiry dates for stat boxes - https://phabricator.wikimedia.org/T170878#3610815 (10elukey) [15:57:24] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Audit users and account expiry dates for stat boxes - https://phabricator.wikimedia.org/T170878#3446584 (10elukey) [16:10:40] OK, elukey we good to go on jmx exporter .deb [16:10:41] it is in apt [16:10:43] https://apt.wikimedia.org/wikimedia/pool/thirdparty/p/prometheus-jmx-exporter/ [16:10:49] nice! [16:11:10] I think that my code review is ready to go then [16:11:29] but there is some work needed for the jmx configuration [16:11:34] aye [16:11:44] two things: [16:11:44] 1) [16:12:25] get rid of the rate metrics, since we can calculate them via prometheus functions.. I tried with the jmx exporter's blacklist but it seems to pick up only Mbeans Object names, and the rate is an attribute (but I could be wrong) [16:12:41] i think that is correct [16:12:57] rate is an attribute (which gets mapped to a prometheus dimension?) [16:13:03] 2) flatten names like kafka_server_brokertopicmetrics_count{name="BytesInPerSec",} (autoproduced by the exported) to something like kafka_server_brokertopicmetrics_BytesInPerSec_count [16:13:12] hmmm [16:13:20] isn't that name=X bit a prometheus dimension? [16:13:21] (all lowercase) [16:13:24] that's a good thing, no? [16:13:44] after a chat with Filippo he'd prefer to flatten the name [16:13:46] or a 'lable' [16:13:47] ? [16:13:48] hm ok [16:13:49] label [16:13:55] https://prometheus.io/docs/practices/naming/ [16:14:45] yeah, i guess most of the mbean attrs aren't really dimensions [16:15:35] we could do something like https://github.com/prometheus/jmx_exporter/blob/master/example_configs/kafka-0-8-2.yml [16:15:41] and explicitly list our metrics [16:15:48] I tried one and it works nicely [16:16:02] everything matching rules is rewritten and published [16:16:04] the rest is not [16:16:34] oo ya those topic, partition labeled dimensinos will be really nice [16:17:04] too bad we have to maintain that still though. ohhh wellll [16:17:14] that will flatten the name as we discussed in 2) [16:17:37] so yeah this is the missing piece in https://gerrit.wikimedia.org/r/#/c/377753/ [16:17:48] Filippo also mentioned that he'll create a prometheus instance on for analytics [16:18:04] (on Monday) [16:18:04] oh a whole instance! ok cool [16:18:18] so my idea is on Monday to come up with a list of rules [16:18:22] review it with you [16:18:27] and then merge if it is fine [16:20:37] a-team: you should all be now able again to release from jenkins [16:20:50] \o/ [16:21:20] ottomata: going afk, lemme know if you do anything on prometheus etc.. [16:21:25] have a good weekend people!! [16:21:39] * elukey off! [16:22:15] ok cool, left a comment, looking good! [17:29:43] 10Analytics-Tech-community-metrics: Zero results shown for certain repositories on Git dashboard (though there has been Git activity) - https://phabricator.wikimedia.org/T175351#3611088 (10Aklapper) Upstreamed as https://gitlab.com/Bitergia/c/Wikimedia/support/issues/3 [17:33:18] 10Analytics-Tech-community-metrics: Empty or incorrect data on Kibana's "Git-Demographics" dashboard - https://phabricator.wikimedia.org/T171240#3611097 (10Aklapper) Upstreamed as https://gitlab.com/Bitergia/c/Wikimedia/support/issues/4 [17:56:50] I'm gonna run out for a walk with bebe, be back later to test wikistats UI stuff [18:05:40] milimetric: lemme know when you are back, woudl love to talk about mw db craziness and history recon [18:24:59] haha, i think this is the first time i'm reading this data lake edits pipeline doc [18:25:00] this is great [18:25:12] " Thanks a lot logging table!" [18:25:12] hahah [19:30:30] 10Analytics: Correct pageview_hourly and derived data for T141506 - https://phabricator.wikimedia.org/T175870#3611263 (10Nuria) >And community members and the public do not have that option and are thus left with the faulty data. This is most certainly not true, any dataset comes with cavetas and issues and this... [19:35:38] wotcha all o/ http://enwp.org/WP:ACTRIAL has gone live on enwp, and we're wondering if its possible to see traffic flow/drop off from [[Wikipedia:New user landing page]] (to evaluate the landing page effectiveness) [19:36:19] is this data collected, and if so how is it available? I'm in the `nda` LDAP group, but I don't think I have access to anything which would possibly hold that data [19:38:14] TheresNoTime, not sure if that's what you're looking for, but in musikanimal's pageview tool, you can see this: https://tools.wmflabs.org/pageviews/?project=en.wikipedia.org&platform=all-access&agent=user&range=latest-20&pages=Wikipedia:New_user_landing_page [19:39:06] Yeah we've got that (which is awesome and super useful!), but if possible we'd love to be able to see where the user goes /next/ (do they create a draft, do they visit the sandbox, or do they just leave etc) [19:52:26] Hi TheresNoTime -- What you're asking for involves tracking, which we don't do usually [19:53:11] TheresNoTime: he only thing I can think of is if people from the Actrial project have implemented some dedicated tracking for new_users over that page using eventLogging [19:53:33] joal: just found out that apparently they have :) [19:53:42] Great :) [19:53:50] Thank you anyway! [19:54:52] :) [20:38:11] milimetric: last update for today: can't manage to find a way to compute "deleted current day" with deletion timestamp - If you want we can brainstorm a bit, but I think I'm gonna go for precomputation [20:53:28] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Audit users and account expiry dates for stat boxes - https://phabricator.wikimedia.org/T170878#3611323 (10DFoy) Please keep access for me as I use this for reviewing Zero traffic along with other country-level research for Global... [21:07:54] 10Analytics, 10Analytics-EventLogging, 10Community-Tech, 10Contributors-Analysis, and 3 others: Visualize page create events for all wikis - https://phabricator.wikimedia.org/T170850#3611355 (10DannyH) [22:59:52] argh, just got back, surely too late for Joseph but I'll brainstorm deleted current day logic. [23:00:32] ottomata, sorry to you too, we can talk Monday at the latest or anytime over the weekend