[07:21:19] nuria: mediawiki adds change tags to edits made with VE, the 2017 wikitext editor, and the mobile editors, but not otherwise. This actually means that we can't distinguish between API edits and 2010 wikitext editor edits, which is a big limitation for metrics. I've been told that the problem is that adding tags for those would create so many tags our infrastructure wouldn't be able to cope, but I could be [07:21:20] wrong about that. [07:22:02] nuria_ ^ [07:54:33] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats 2.0: Add statistics for the geographical origin of the contributors - https://phabricator.wikimedia.org/T188859#4022948 (10Neil_P._Quinn_WMF) >>! In T188859#4022034, @ezachte wrote: > @Pamputt the url you give is about views not viewers. This is so often mix... [07:55:52] 10Analytics-Kanban: Make aggregate data on editors per country per wiki publicly available - https://phabricator.wikimedia.org/T131280#2161972 (10Neil_P._Quinn_WMF) [08:17:46] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Upgrade eventlogging servers to Stretch - https://phabricator.wikimedia.org/T114199#4022965 (10elukey) [08:17:49] 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar), and 2 others: Deprecation of mw.errors.* metrics - https://phabricator.wikimedia.org/T188749#4022964 (10elukey) [08:35:11] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Upgrade eventlogging servers to Stretch - https://phabricator.wikimedia.org/T114199#4022993 (10elukey) Some things that would be nice to complete before the migration: * https://gerrit.wikimedia.org/r/#/c/41588... [08:45:15] all right I think I got all the moving gears on eventlog1001 :D [08:45:52] I'd prefer to wait for those things to be deprecated/removed before attempting the migration, since we are not in a incredible hurry [08:46:50] Cool elukey ! good morning B-) [08:49:39] morningggg [08:52:54] stat1004 died yesterday for some reason [08:53:02] had to manually powercycle it.. [08:53:03] wow :( [08:53:09] any idea of wh [08:53:11] the good thing is that it runs the new kernel :) [08:53:16] hehehe :) [08:53:36] not really, I think it was a hw froze event, didn't find much traces in logs and console [08:54:27] elukey: https://grafana-admin.wikimedia.org/dashboard/file/server-board.json?refresh=1m&orgId=1&var-server=stat1004&var-network=eth0&from=now-2d&to=now [08:54:57] elukey: my 2 cents guess: diego tried his experiment on dumps ;) [08:55:18] hahahahah could be yes [08:55:20] elukey: sorry for the admin link :( [08:55:22] I saw Mar 4 13:04:01 stat1004 cron[753]: (CRON) error (can't fork) [08:55:34] everything was frozen yesterday [08:55:47] that's uncool [08:55:53] dsaez: hello :) any chance that you did some experiment on stat1004 yesterday? [08:56:02] joal: it happens, no big deal :) [08:56:06] yup [08:56:38] it kinda makes my life easier in upgrading the kernels :P [08:59:30] 10Analytics-Kanban, 10Cloud-Services: Provide mediawiki history data to Cloud Services users - https://phabricator.wikimedia.org/T169572#4023042 (10Neil_P._Quinn_WMF) [09:49:10] just merged eventloggingctl for systemd [09:49:19] should be almost the same as the upstart version [09:49:29] so the transition for whover will do ops week should be easier [09:57:59] joal: do you know why json refine fails on SearchSatisfaction, but (afaict) there is no schema for it in meta? [09:58:05] am I looking in the wrong place? [09:59:02] ah weird it is in eventlogging_valid_mixed_schema_blacklist (puppet) [09:59:39] elukey: I'm actually not sure what this balcklist is for :S [10:00:13] I think that they don't go in valid-mixed so the mysql consumer will not pick them up [10:00:30] elukey: Right, that was what I thought [10:00:36] elukey: so, it's in hadoop only [10:02:15] but I don't find the schema [10:05:08] Mwarf :( [10:05:17] elukey: give a minute, I'll try to help [10:06:20] joal: probably not urgent, it was only rambling :) [10:06:30] please keep going with your work [10:08:59] in the meantime, rebooting some analytics workers [10:33:43] drain+reboot of an1046-49 [10:57:23] !log Killing failing Mediawiki-History job for 2018-03 [10:57:24] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:57:52] !log Relaunch Mediawiki-history job manually from spark2 to see if new versions helps [10:57:54] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:03:36] ah wasn't me rebooting! nice :) [11:04:18] joal: iirc you told me to alert you beforehand when an1051 was going to be rebooted [11:04:27] I am planning to do the 50 series this afternoon [11:04:46] elukey: Thank you for having reminded :) [11:05:06] elukey: An almost clean patch has been made for streaming job, so we are safe [11:05:11] elukey: You can go for it :) [11:06:42] joal: ack! [11:07:04] I am sending emails to screen/tmux owners for stat100[56], planning to reboot them on Wed [11:07:15] good for me elukey [11:28:39] 10Analytics, 10Operations, 10Traffic: Update documentation for "https" field in X-Analytics - https://phabricator.wikimedia.org/T188807#4023315 (10MoritzMuehlenhoff) p:05Triage>03Normal [11:31:12] elukey: hdfs is having difficulties I think :( [11:31:54] elukey: simple read commands take ages, with connection issues [11:37:14] joal: even now? Might have been due to your commands reaching down nodes [11:37:23] I don't see anything weird in metrics [11:37:59] elukey: nope, super good now [11:38:15] elukey: maybe it was because of restarts? [11:38:17] eird [11:38:51] elukey: I'm gonna push for a spark2 move - mediawiki-history job is really faster [11:38:58] \o/ [11:41:02] joal: if everything is ok I'd leave for lunch + errand [11:41:16] all hosts are up and running [11:41:22] Awesome :) Thanks elukey :) [11:41:42] see you later!! [11:41:45] * elukey afk! [11:41:47] later! [13:10:51] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jan-Mar-2018): Adjust to Grimoirelab / Bitergia moving to GitLab - https://phabricator.wikimedia.org/T171290#4023559 (10Aklapper) [13:10:53] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jan-Mar-2018): Adjust to Grimoirelab / Bitergia moving to GitLab - https://phabricator.wikimedia.org/T171290#3459901 (10Aklapper) 05Open>03Resolved Other things happened in the meantime, e.g. Grimoirelab having moved under the umbrella of [[ https:... [14:23:05] 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar), and 2 others: Deprecation of mw.errors.* metrics - https://phabricator.wikimedia.org/T188749#4023747 (10fgiunchedi) >>! In T188749#4021055, @elukey wrote: > Thanks @Krinkle! @fgiunchedi I think we are ready to go, what do you... [14:29:47] Good morning! While I await review of https://gerrit.wikimedia.org/r/#/c/415798/, I've been reading about Dashiki so I can be ready to move on to graphing the data. [14:29:51] Initially, it looked like the tabs layout with type dygraphs-timeseries would be appropriate for the data. [14:30:02] However, at https://wikitech.wikimedia.org/wiki/Analytics/Tutorials/Dashboards#Config_for_the_tabs_layout, it states, “Tip: each column in your report (except the first that holds the date) will become a line in the chart.” [14:30:13] In the data I am producing, each row is of the form “date label count” with multiple rows for each data. This is because the labels cannot be determined a priori. [14:30:30] For example: [14:30:50] 2018-03-01        Apache        42897 [14:30:55] 2018-03-01        nginx        3529 [14:31:01] 2018-03-01        LiteSpeed        985 [14:31:07] 2018-03-01        Microsoft-IIS        306 [14:31:11] 2018-03-01        lighttpd        234 [14:31:16] 2018-03-01        Other        204 [14:31:22] Is there a Dashiki format for graphing data over time that is appropriate for multiple rows of data for the same date? [14:32:40] Hi CindyCicaleseWMF - I think the right persons to answer your question would be milimetric and mforns[m] [14:33:57] hi [14:34:07] I was reading, just had to reboot [14:34:15] 10Analytics, 10Operations, 10Traffic: Update documentation for "https" field in X-Analytics - https://phabricator.wikimedia.org/T188807#4023770 (10Tbayer) Very informative, thanks @BBlack! So I understand that these cases would explain the 1% of requests in T188807#4021737 that have NULL for `x_analytics_map... [14:35:02] 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar), and 2 others: Deprecation of mw.errors.* metrics - https://phabricator.wikimedia.org/T188749#4023773 (10elukey) [14:35:12] hi CindyCicaleseWMF, so, you could visualize it as a pie chart type thing, like this: https://analytics.wikimedia.org/dashboards/browsers/#all-sites-by-os/os-family-and-major-hierarchical-view [14:35:50] let me remember if we built in some pivoting feature or if you have to pivot it yourself [14:35:56] That would be great for a given point in time, but I'd like to see over time how each value changes. [14:36:07] yeah, of course [14:39:13] CindyCicaleseWMF: yeah, we talked about it but we haven't implemented it yet. But we could do it pretty easily. The main problem is there's no way to pivot in mysql, otherwise you could do it yourself in the query, right? [14:40:22] CindyCicaleseWMF: if you have a fixed number of things you want to count, you can always do the gross manual pivoting trick like this: [14:40:24] Well, the problem is that the columns could change over time. So, I couldn't know when the report is first generated what the columns need to be. [14:40:58] oh... :) yeah, postgresql would just let you pivot without knowing, and reportupdater is ok with that, if you add or remove columns, it adjusts accordingly [14:42:18] ok, then, we'd have to add a pivotOnIndex: 1 option, that will be useful for other folks. How soon do you need this CindyCicaleseWMF? [14:42:21] Hmm, so maybe it would work with reportupdater? The MediaWiki version query is the best example. When a new MediaWiki version is released, a new column would need to get added in the .tsv file. How does reportupdater handle that? Doesn't it just keep appending data? [14:43:30] It is not urgent, but it would be great to be able to publish the aggregated pingback data "soon". We've been promising it for a year, and there is enough data now to be interesting. [14:43:50] yeah, reportupdater will automatically recognize new columns added or old columns removed and update old history to match new formats. So if you start with A, B, C and your new query outputs A, Y, X, C, then reportupdater would remove column B from old output and add Y and X as NULL all the way back to the beginning [14:44:42] ok, I'll make a task and ping you on it, and bring it up with the team today in grooming. I'll let you know what we do, we just have a few pressing things to finish this week. [14:45:54] Excellent! Thanks! Also, merge of https://gerrit.wikimedia.org/r/#/c/415798/ would be great if you all think it is ready. [14:46:48] 10Analytics, 10Analytics-Dashiki: Add pivot parameter to tabular layout graphs {lama} - https://phabricator.wikimedia.org/T126279#2009951 (10Milimetric) [15:00:24] !log rerun mediacounts-load-wf-2018-3-5-9 [15:00:25] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:16:26] 10Analytics-Kanban, 10Operations, 10ops-eqiad: DIMM errors for analytics1062 - https://phabricator.wikimedia.org/T187164#4023867 (10Cmjohnson) @elukey I will need to shutdown the server down and move the dimm around. Let me know when it's safe to do this [15:23:50] 10Analytics-Kanban, 10Operations, 10ops-eqiad: DIMM errors for analytics1062 - https://phabricator.wikimedia.org/T187164#4023874 (10elukey) >>! In T187164#4023867, @Cmjohnson wrote: > @elukey I will need to shutdown the server down and move the dimm around. Let me know when it's safe to do this The server... [15:37:36] 10Analytics, 10Operations, 10Traffic: Update documentation for "https" field in X-Analytics - https://phabricator.wikimedia.org/T188807#4023886 (10BBlack) >>! In T188807#4023770, @Tbayer wrote: > And is it correct to assume besides those HTTP --> HTTPS redirects, there are other cases where we send a 301 rep... [15:42:04] rebooting analtics1055->57 [16:01:17] (03CR) 10Milimetric: "mentioned this trick, it's optional, and we'll get you an estimate on when we can pivot in Dashiki after grooming today." (031 comment) [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/415798 (https://phabricator.wikimedia.org/T152222) (owner: 10Cicalese) [16:24:04] 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Alarm when /mnt/hdfs is mounted but showing no files/dirs in there - https://phabricator.wikimedia.org/T187073#4024045 (10elukey) >>! In T187073#4023938, @ArielGlenn wrote: > The current changeset would check the mount point at every puppet run, i.e. eve... [16:26:44] 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Alarm when /mnt/hdfs is mounted but showing no files/dirs in there - https://phabricator.wikimedia.org/T187073#4024052 (10ArielGlenn) >>! In T187073#4024045, @elukey wrote: > Definitely, one alternative could be to do something like ls -l /mnt/hdfs and... [16:36:54] 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Alarm when /mnt/hdfs is mounted but showing no files/dirs in there - https://phabricator.wikimedia.org/T187073#4024079 (10elukey) The other issue is that sometimes users might have fd open to /mnt/hdfs/something preventing a clean umount (without force).... [16:51:17] 10Analytics, 10Operations: setup/install eventlog1002.eqiad.wmnet - https://phabricator.wikimedia.org/T185667#4024131 (10elukey) [16:51:20] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Upgrade eventlogging servers to Stretch - https://phabricator.wikimedia.org/T114199#4024130 (10elukey) [16:53:37] 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10Goal, 10Services (doing): FY17/18 Q3 Program 8 Services Goal: Migrate two high-traffic jobs over to EventBus - https://phabricator.wikimedia.org/T183744#4024137 (10mobrovac) [16:53:42] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 4 others: Select candidate jobs for transferring to the new infrastucture - https://phabricator.wikimedia.org/T175210#4024138 (10mobrovac) [16:53:46] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 5 others: Switch cdnPurge to Kafka - https://phabricator.wikimedia.org/T188540#4024136 (10mobrovac) [16:54:26] 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10Goal, 10Services (doing): FY17/18 Q3 Program 8 Services Goal: Migrate two high-traffic jobs over to EventBus - https://phabricator.wikimedia.org/T183744#3862480 (10mobrovac) [16:54:33] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 4 others: Switch cdnPurge to Kafka - https://phabricator.wikimedia.org/T188540#4011588 (10mobrovac) 05Open>03Resolved The `cdnPurge` has been successfully moved over to EventBus. Resolving. [17:02:42] ping joal, grosssskin [17:05:54] 10Analytics, 10Analytics-Dashiki: Add pivot parameter to tabular layout graphs - https://phabricator.wikimedia.org/T126279#4024173 (10fdans) [17:16:52] (03CR) 10Milimetric: "Ok, we prioritized this for next quarter, unfortunately we can't fit it into this quarter, we're swamped. Let me know if you want to do t" [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/415798 (https://phabricator.wikimedia.org/T152222) (owner: 10Cicalese) [17:22:39] 10Analytics, 10Operations, 10Traffic: Investigate and fix odd uri_host values - https://phabricator.wikimedia.org/T188804#4024228 (10fdans) [17:25:29] neilpquinn: got it [17:27:13] 10Analytics-Kanban, 10Analytics-Wikistats: Routing code allows invalid routes - https://phabricator.wikimedia.org/T188792#4024240 (10fdans) p:05Triage>03Unbreak! [17:36:40] PROBLEM - Hadoop NodeManager on analytics1059 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager [17:37:00] PROBLEM - Hadoop NodeManager on analytics1058 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager [17:37:21] PROBLEM - Hadoop NodeManager on analytics1060 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager [17:37:57] this is me rebooting, sorry for the spam [17:46:04] 10Analytics, 10Operations, 10Traffic: Investigate and fix odd uri_host values - https://phabricator.wikimedia.org/T188804#4020291 (10BBlack) The bottom line is that the value of `uri_host` is entirely up to the client, and therefore subject to client-side stupidity. It's legal (in all protocol senses) for a... [17:46:39] 10Analytics: Changes to map projection in wikistats - https://phabricator.wikimedia.org/T188927#4024327 (10Nuria) [17:47:01] 10Analytics: Changes to map projection in wikistats - https://phabricator.wikimedia.org/T188927#4024341 (10Nuria) ping @ezachte [17:53:50] RECOVERY - Hadoop NodeManager on analytics1059 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager [17:54:01] 10Analytics: Intervals for data arround pageviews in wikistats maps - https://phabricator.wikimedia.org/T188928#4024359 (10Nuria) [17:54:01] RECOVERY - Hadoop NodeManager on analytics1058 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager [17:55:14] 10Analytics, 10EventBus, 10Services (doing), 10User-Elukey: ChangeProp workers die if they can't connect to redis - https://phabricator.wikimedia.org/T179684#4024382 (10elukey) [17:57:50] elukey: still reboots to happen on hadoop for this evening? [18:00:56] joal: if you want I can stop and start again tomorrow morning, any issue? [18:01:17] elukey: my spark job is a bit sensitive - But for now you can go, the tricky part happen later :) [18:01:48] ah! Ok I am stopping now, no rush, will restart tomorrow morning :) [18:05:09] (03PS23) 10Joal: Upgrade scala to 2.11.7 and Spark to 2.2.1 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/348207 [18:05:11] (03PS4) 10Joal: [WIP] Add by-wiki stats to MediawikiHistory job using new MapAccumulator [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/415255 (https://phabricator.wikimedia.org/T155507) [18:05:34] Heya mforns - Do you mind having a look at --^, checking it goes well with what we discussed? [18:06:01] joal, sure [18:09:17] 10Analytics, 10EventBus, 10Services (doing), 10User-Elukey: ChangeProp workers die if they can't connect to redis - https://phabricator.wikimedia.org/T179684#3732899 (10mobrovac) For the record, this is still happening. Here's the last instance: a worker on `scb1001` was in charge of the `RecordLintJob` to... [18:09:56] 10Analytics, 10EventBus, 10Services (doing), 10User-Elukey: Kafka sometimes misses to rebalance topics properly - https://phabricator.wikimedia.org/T179684#4024457 (10mobrovac) [18:12:08] joal, looks good overall, do you want me to carefully CR the whole code now? [18:13:50] nuria_: I forgot to mention that over the weekend bohrium died again, so there is the chance that the invalidation might not have been completely processed [18:14:26] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Add by-wiki stats to MediawikiHistory job using new MapAccumulator [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/415255 (https://phabricator.wikimedia.org/T155507) (owner: 10Joal) [18:21:57] !log restart webrequest-load-wf-text-2018-3-5-16 via Hue (failed due to reboots) [18:21:58] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:22:20] !log restart webrequest-load-wf-upload-2018-3-5-16 via Hue (failed due to reboots) [18:22:21] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:25:23] elukey: should we do crons every couple hours rather than once a day? [18:26:23] nuria_: we could do it, I'll try to set it up tomorrow [18:26:42] elukey: i can do that and sumit a puppet patch [18:27:11] 10Analytics-Kanban, 10Wikipedia-iOS-App-Backlog: iOS traffic data is not available on Piwik since Feb 20, 2018 - https://phabricator.wikimedia.org/T188559#4024574 (10Nuria) 05Resolved>03Open [18:27:34] 10Analytics-Kanban, 10Wikipedia-iOS-App-Backlog: iOS traffic data is not available on Piwik since Feb 20, 2018 - https://phabricator.wikimedia.org/T188559#4012140 (10Nuria) 05Open>03Resolved [18:27:56] nuria_: sure [18:30:15] * elukey off! [18:39:22] (03PS7) 10Cicalese: Add ReportUpdater queries for MediaWiki pingback data. [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/415798 (https://phabricator.wikimedia.org/T152222) [18:43:41] (03CR) 10Cicalese: "OK, thanks! Please review again. After examining the data, I have gone back and manually pivoted all but one of the queries (version.sql)." [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/415798 (https://phabricator.wikimedia.org/T152222) (owner: 10Cicalese) [18:44:10] (03PS8) 10Cicalese: Add ReportUpdater queries for MediaWiki pingback data. [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/415798 (https://phabricator.wikimedia.org/T152222) [18:58:09] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 4 others: Select candidate jobs for transferring to the new infrastucture - https://phabricator.wikimedia.org/T175210#4024653 (10Pchelolo) [19:01:37] 10Analytics: piwik archive job should happen more frequently - https://phabricator.wikimedia.org/T188939#4024657 (10Nuria) [19:21:46] (03PS5) 10Joal: [WIP] Add by-wiki stats to MediawikiHistory job using new MapAccumulator [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/415255 (https://phabricator.wikimedia.org/T155507) [19:22:28] mforns: I think the path above is ready for thorough review - Do you want to do it now, or do you prefer me adding pages and revisions firs? [19:35:44] joal, I will do a thorough review now and give you a +1, but do not wait for me please, don't want to block you, I don't think the code needs any refactor! [19:36:14] mforns: I am going to lift some graphics from your datos conscientes presentation [19:36:53] sure! do you want the google docs? [19:37:02] or you're ok with the pdf? [19:54:00] (03CR) 10Milimetric: "Everything looks great, I'm just double checking one detail on machine.sql. Also, I'm sorry a limitation in our tool made you write these" (031 comment) [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/415798 (https://phabricator.wikimedia.org/T152222) (owner: 10Cicalese) [19:59:18] (03CR) 10Cicalese: Add ReportUpdater queries for MediaWiki pingback data. (031 comment) [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/415798 (https://phabricator.wikimedia.org/T152222) (owner: 10Cicalese) [20:00:51] (03PS9) 10Cicalese: Add ReportUpdater queries for MediaWiki pingback data. [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/415798 (https://phabricator.wikimedia.org/T152222) [20:02:37] (03CR) 10Cicalese: "OK, maybe it's ready now? Thanks for all of your help with this! I really thought I could get it right the first time, but I appreciate th" [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/415798 (https://phabricator.wikimedia.org/T152222) (owner: 10Cicalese) [20:09:06] mforns: just took couple images, we might not end up using them [20:12:04] fdans: do you know how to put the wikistats code on teh canary site? [20:17:36] nuria: (sorry having dinner) yep! already have it on transmit for sync [20:20:11] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 3 others: Create an LVS endpoint for videoscalers - https://phabricator.wikimedia.org/T188947#4024921 (10Pchelolo) p:05Triage>03Normal [20:20:23] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 2 others: Create an LVS endpoint for videoscalers - https://phabricator.wikimedia.org/T188947#4024934 (10Pchelolo) [20:23:35] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 3 others: Create an LVS endpoint for jobrunners on videoscalers - https://phabricator.wikimedia.org/T188947#4024939 (10mobrovac) [20:23:54] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 4 others: Create an LVS endpoint for jobrunners on videoscalers - https://phabricator.wikimedia.org/T188947#4024921 (10mobrovac) [20:59:42] 10Analytics, 10Analytics-EventLogging, 10Performance-Team (Radar), 10Readers-Web-Backlog (Tracking): Make it easier to enable EventLogging's debug mode - https://phabricator.wikimedia.org/T188640#4025027 (10Imarlier) [21:15:08] fdans: please try commit also a gerrit change when you have something you think is good enough for canary so others can comment with ample time [21:18:10] 10Analytics-Kanban, 10Patch-For-Review: piwik archive job should happen more frequently - https://phabricator.wikimedia.org/T188939#4025080 (10Nuria) [21:18:48] 10Analytics-Kanban, 10Patch-For-Review: piwik archive job should happen more frequently - https://phabricator.wikimedia.org/T188939#4024657 (10Nuria) a:03Nuria [21:19:33] 10Analytics-Kanban, 10Patch-For-Review: piwik archive job should happen more frequently - https://phabricator.wikimedia.org/T188939#4024657 (10Nuria) [21:25:45] super confused... none of this cu_changes data / schema makes sense [21:26:06] it's like the schema and documentation is more of a guideline that mediawiki code seems to follow ... sometimes [21:28:04] milimetric: You are not ALONE [21:28:24] nuria_: I'm happy to brainbounce, but I don't know if I know enough yet [21:34:08] I’ve been reading https://wikitech.wikimedia.org/wiki/Analytics/Systems/Dashiki and https://wikitech.wikimedia.org/wiki/Analytics/Tutorials/Dashboards and am trying to figure out what the required pieces are to create a dashboard for .tsv files generated by reportupdater. [21:34:18] I’ve been tracing through two examples: page creations and user agent breakdowns. [21:34:29] For page creations, I see that there is a configuration page at https://meta.wikimedia.org/wiki/Config:Dashiki:PageCreations plus site configuration at https://github.com/wikimedia/analytics-dashiki/blob/master/config.yaml in order to create https://page-creation.wmflabs.org. [21:34:37] For user agent breakdowns, I see that there is a configuration page at https://meta.wikimedia.org/wiki/Config:Dashiki:SimpleRequestBreakdowns in order to create https://analytics.wikimedia.org/dashboards/browsers/#all-sites-by-os. [21:34:42] CindyCicaleseWMF: it's just a config page, the rest we'll take care of [21:34:43] I haven’t found a config.yaml file for the user agent breakdowns dashboard. [21:35:20] the config.yaml has all the user dashboards, the other dashboards are our team's so we have separate config [21:35:21] OK, so if I create Config:Dashiki:Pingback, I'm good to go? That would be great! [21:35:34] CindyCicaleseWMF: yeah, and ping us, we'll need to finish manually but it's easy [21:35:40] 10Analytics-Kanban: Adapt TableChart to compact size - https://phabricator.wikimedia.org/T188952#4025114 (10fdans) [21:36:30] CindyCicaleseWMF: did milimetric explained that the data will not be pivoted? [21:36:44] Great! Would the dashboard appear with its own URL like page-creations.wmflabs.org? [21:37:02] nuria: I pivoted the data with his help [21:37:11] CindyCicaleseWMF: you can tell us to either put it on an existing domain in a subfolder or give a new domain [21:37:33] So, do I need to create a VM to host it? [21:37:48] CindyCicaleseWMF: yes, you can test 1st on localhost by downloading dashiki's code and running dashiki locally against your config (but this is easy or hard depending how familiar you are with frontend stuff) https://wikitech.wikimedia.org/wiki/Analytics/Tutorials/Dashboards#Test_locally_2 [21:38:00] CindyCicaleseWMF: no, we have vms to host dashboards [21:38:18] CindyCicaleseWMF: example: https://edit-analysis.wmflabs.org/multimedia-health/#projects=enwiki,dewiki,commonswiki/metrics=Uploads [21:38:46] Yes, I saw the info about testing locally. I was hoping to avoid creating and provisioning a VM :-) But, I can do so. [21:38:48] CindyCicaleseWMF: yeah, all the rest of the details are taken care of, you really just make a config page and ping us with the domain you want [21:39:05] Oh, then you can create the VM at that domain? [21:39:29] CindyCicaleseWMF: no need to create any vms [21:39:30] I guess I'm not understanding one of you. [21:39:30] CindyCicaleseWMF: we host all of them on the same VM with different vhosts, but yea [21:39:38] 10Analytics-Kanban: Limit length of table and add "Load more rows" button - https://phabricator.wikimedia.org/T188953#4025130 (10fdans) [21:39:40] Ah, OK, great. [21:39:44] Much better. [21:39:47] CindyCicaleseWMF: testing locally requires no vm [21:40:14] CindyCicaleseWMF: it is just javascript, there is no server code [21:40:26] OK, gotcha. [21:40:36] CindyCicaleseWMF: locally means your desktop [21:40:58] Right. I wasn't asking about testing locally but about hosting the final dashboard. Testing locally is not a problem. [21:41:43] But, as I understand it, once I've created the config page on meta and tested locally, once I ping you all, you'll take care of creating the actual dashboard. [21:43:07] I also need to give you a domain, like pingback.wmflabs.org. [21:43:19] 10Analytics-Kanban, 10Analytics-Wikistats: SEO-friendly HTML titles for Wikistats 2.0 - https://phabricator.wikimedia.org/T182718#4025141 (10Nuria) Ok, piwik supports tracking client sdie urls: https://github.com/matomo-org/matomo/issues/7998 [21:43:33] That sounds pretty simple. Thanks. [21:44:09] One last question: for the format parameter, I see kmb and percent being the possible values. What is kmb? Am I going to feel foolish for not knowing when you tell me? [21:44:52] CindyCicaleseWMF: yes, give us a domain that you like [21:45:13] CindyCicaleseWMF: no, it just not obvious k = kilobytes [21:45:24] CindyCicaleseWMF: so the units you owuld use to measure bytes [21:45:48] CindyCicaleseWMF: makes sense? [21:46:38] I think so. The numbers I'll be plotting are counts, so not percent or kmb. Do I just leave off the format parameter? I noticed that it is not always specified. [21:51:17] milimetric: do you have a second to talk about CSS? [21:51:25] I sound like a jehova's witness [21:51:32] * milimetric runs away [21:51:39] wait! you said CSS! Let's do it [21:51:45] omw [22:05:01] jajaja