[00:06:43] Analytics-EventLogging, ArchCom-RfC, Discovery, Graphs, and 9 others: RFC: Use YAML instead of JSON for structured on-wiki content - https://phabricator.wikimedia.org/T147158#2691375 (Yurik) [00:56:39] Analytics-Kanban, MediaWiki-extensions-WikimediaEvents, Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016), Patch-For-Review, WMF-deploy-2016-10-04_(1.28.0-wmf.21): EL alarms raw/validated 20160926 - https://phabricator.wikimedia.org/T146674#2691534 (Etonkovidova) Re-checked vagrant logs... [00:57:11] Analytics-Kanban, MediaWiki-extensions-WikimediaEvents, Collab-Team-Q1-July-Sep-2016, Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016), and 2 others: EL alarms raw/validated 20160926 - https://phabricator.wikimedia.org/T146674#2691535 (Etonkovidova) [01:13:15] Analytics-EventLogging, ArchCom-RfC, Discovery, Graphs, and 9 others: RFC: Use YAML instead of JSON for structured on-wiki content - https://phabricator.wikimedia.org/T147158#2691573 (Yurik) I just moved the examples into description, as they represent the reason why in some cases YAML-like struc... [03:31:11] Analytics-EventLogging, ArchCom-RfC, Discovery, Graphs, and 9 others: RFC: Use YAML instead of JSON for structured on-wiki content - https://phabricator.wikimedia.org/T147158#2691615 (matmarex) Hjson's quoteless strings mildly conflict with wikitext links and templates. For example, this is a syn... [03:56:55] Analytics-EventLogging, ArchCom-RfC, Discovery, Graphs, and 9 others: RFC: Use YAML instead of JSON for structured on-wiki content - https://phabricator.wikimedia.org/T147158#2691631 (Yurik) Well, this isn't exactly a conflict with Wiki markup - it is simply that a quote-less string cannot begin... [06:42:45] Analytics-EventLogging, ArchCom-RfC, Discovery, Graphs, and 9 others: RFC: Use YAML instead of JSON for structured on-wiki content - https://phabricator.wikimedia.org/T147158#2683091 (Base) @Yurik, I must note that Graph is super uber mega ultra difficult as it is, with very poor documentation. I... [07:16:27] Analytics-EventLogging, DBA, ImageMetrics: Drop EventLogging tables for ImageMetricsLoadingTime and ImageMetricsCorsSupport - https://phabricator.wikimedia.org/T141407#2691729 (Marostegui) I am tempted to close this ticket as the only table that is still in use (39 records now) is: ImageMetricsCorsSu... [10:16:46] Analytics-Tech-community-metrics: Deployment of Mediawiki panels - https://phabricator.wikimedia.org/T138006#2692129 (Lcanasdiaz) >>! In T138006#2681684, @Aklapper wrote: > Looks like this vanished again. I don't see any wiki or MediaWiki panel anymore among our Git, Gerrit and Mailing list ones. > Hence reo... [10:36:10] Analytics-Tech-community-metrics, Developer-Relations (Oct-Dec-2016): Allow AKlapper to access https://wikimedia.biterg.io/edit/ - https://phabricator.wikimedia.org/T144704#2692167 (Lcanasdiaz) >>! In T144704#2607839, @Aklapper wrote: > Or... actually I think I mean https://wikimedia.biterg.io/edit inste... [10:37:46] Analytics-Tech-community-metrics, Developer-Relations (Oct-Dec-2016): Allow AKlapper to access https://wikimedia.biterg.io/edit/ - https://phabricator.wikimedia.org/T144704#2692169 (Lcanasdiaz) >>! In T144704#2692167, @Lcanasdiaz wrote: >>>! In T144704#2607839, @Aklapper wrote: >> Or... actually I think... [11:21:41] Analytics-Tech-community-metrics: Kibana's Mailing List data sources are outdated - https://phabricator.wikimedia.org/T146632#2692195 (Lcanasdiaz) a:Lcanasdiaz There is a problem with the data collection, look: https://wikimedia.biterg.io:443/goto/0cdea3035ff546ab89518b14558c3a38 Let's find out why .. [11:22:54] Analytics-Tech-community-metrics: Two "Submitters" widgets in Kibana show no data: "Could not locate that index-pattern-field (id:project)" - https://phabricator.wikimedia.org/T146629#2692198 (Lcanasdiaz) a:Lcanasdiaz [11:44:55] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [30.0] [11:50:03] RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK: OK: Less than 20.00% above the threshold [20.0] [12:01:53] Analytics-Tech-community-metrics, Developer-Relations (Oct-Dec-2016): Go through default Kibana widgets; decide which ones are not relevant for us and remove them - https://phabricator.wikimedia.org/T147001#2692265 (Aklapper) [12:01:55] Analytics-Tech-community-metrics, Developer-Relations (Oct-Dec-2016), Documentation: Create basic Kibana (dashboard) documentation for admins - https://phabricator.wikimedia.org/T145929#2692266 (Aklapper) [12:01:57] Analytics-Tech-community-metrics, Developer-Relations (Oct-Dec-2016): Allow AKlapper to access https://wikimedia.biterg.io/edit/ - https://phabricator.wikimedia.org/T144704#2692263 (Aklapper) Open>Resolved Temporary pass works for me, I'd say - I can log in and the password isn't 1234. :P Thank... [12:07:19] Analytics-Tech-community-metrics, Developer-Relations (Oct-Dec-2016), Documentation: Create basic Kibana (dashboard) documentation for admins - https://phabricator.wikimedia.org/T145929#2692275 (Aklapper) p:Low>Normal [12:12:47] is someone available who has access to the compute clusters? I am having a security question here [12:13:21] Hi Alex_____, what would like to know? [12:13:55] is it possible that the rsa host key for bast3001 changed today? [12:14:30] yes we have re-installed the OS today, godog is about to finish the process :) [12:15:15] alright thanks :) I was getting a bit nervous about it [12:15:26] i.e. I can safely update the key [12:15:52] :) [12:16:55] At what time would you estimate will I be able to login through key authentication again? :o [12:27:16] I think that it should be ready soon, maybe 10/20 mins? [12:39:40] Alright, thanks for the info :) have a g'day [12:41:49] you too! [13:42:30] Analytics-Kanban: Examine puppet code for Event Logging and make sure monitoring is using the best counts - https://phabricator.wikimedia.org/T147321#2692566 (Milimetric) [13:44:49] Analytics-Kanban, MediaWiki-extensions-WikimediaEvents, Collab-Team-Q1-July-Sep-2016, Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016), and 2 others: EL alarms raw/validated 20160926 - https://phabricator.wikimedia.org/T146674#2692567 (Milimetric) @Etonkovidova: I'm wondering, is the code... [14:10:28] Analytics-Kanban: Examine puppet code for Event Logging and make sure monitoring is using the best counts - https://phabricator.wikimedia.org/T147321#2692636 (Milimetric) Ok, so currently, according to the eventlogging processes set up by [1]: 1. events come in on raw kafka topics, server and client side 2.... [14:10:45] elukey: you might want to check out that last comment ^ about EL monitoring [14:10:53] it wasn't quite doing what we thought it was :) [14:11:47] mforns / nuria_: shall I deploy the dashiki code? Any reservations? [14:12:16] milimetric, not on my side :] [14:13:03] mforns: ok, related question: [14:13:11] sure :) [14:13:30] for a while I've been wanting to slim down the sessions metric so it doesn't completely kill the UI rendering that sunburst [14:13:39] milimetric: do you have any idea if /a/wikistats_git/pageviews_reports/bin/stat1-cron-script.sh on stat1002 still needs to be executed in cron? [14:13:45] the one on the compare dashboard [14:14:04] elukey: no, I don't know if that's new erik code or old erik code [14:14:45] Erik told me that the code was written from Stefan Petrea a long time ago and he doesn't know if it is needed or not (since it predates pageviews on hadoop_ [14:14:48] milimetric, yes, didn't we want to make it weekly? [14:17:22] mforns: yeah, weekly, by like aggregating the existing file and then changing the query [14:17:29] milimetric, aha [14:17:37] mforns: that would work, right? [14:18:10] milimetric, yes! the alternative being letting RU recalculate everything... But this would take some time... [14:18:26] forever, right :) 'cause it's for a bunch of wikis :) [14:18:33] yes [14:18:34] oh god, we have to do this with like 800 files [14:18:45] now i remember why i've been scared to do it [14:18:46] we need a script right [14:18:52] * milimetric plugs his nose and gets to work [14:19:09] yeah, a script to apply to that whole directory [14:19:20] aha [14:21:02] milimetric, if RU had that feature that we discussed the other day: the one that was already implemented in Wikimetrics, of detecting ranges of empty data, it would help, but... [14:21:08] mforns: I forget, does weekly generate data on Sunday always or how does that work [14:21:42] milimetric, yes, starts on sunday 00:00, finishes on saturday 23:59:59 [14:22:10] ok, so the script has to take all dates in that range and roll them up to Sunday 00:00 [14:22:29] omg, some of these files are tens of MB [14:24:19] mforns: finding empty data wouldn't help, the existing data has to be aggregated [14:24:39] 163MB in total [14:24:42] ouch :) [14:25:22] milimetric, hehe, the script will be tired after all those aggregations :] [14:25:47] yeah, it'll need gatorade [14:25:57] milimetric, I meant, with that feature we could just delete the data, and let RU fill the data in with just a single query per wiki. [14:25:59] xDDD [14:27:31] oh yeah, true [14:27:56] man, this cafe is evil, has no power outlet so I might try standup on my ipad [14:33:54] milimetric, hehe [14:34:03] I found 5 candidates without review [14:34:10] and I finished them [14:34:55] nice, perfect, we're done [14:35:08] Yay :) [14:35:38] o/ [14:37:52] Analytics: Varnishkafka testing framework - https://phabricator.wikimedia.org/T147432#2692790 (elukey) [14:41:52] I'm sitting next to bankers that are talking about how they can squeeze private user data for marketting data and sell it [14:43:25] xDDD [14:43:51] Analytics: Better Compiler warnings in Makefile - https://phabricator.wikimedia.org/T147436#2692853 (elukey) [14:44:01] Analytics: Varnishkafka testing framework - https://phabricator.wikimedia.org/T147432#2692790 (elukey) p:Triage>Normal [14:45:04] milimetric, nuria_, we'll have 104 candidates for second round + 25 that have already started hr screening [14:45:26] Analytics: Valgrind tutorial for periodical mem usage reviews - https://phabricator.wikimedia.org/T147438#2692888 (elukey) [14:47:37] Analytics: Evaluate a unit testing framework and add tests for the formatter function - https://phabricator.wikimedia.org/T147440#2692917 (elukey) [14:48:40] Analytics: Find a strategy for integration testing - https://phabricator.wikimedia.org/T147442#2692947 (elukey) [14:49:03] nuria_: Hola! Just created https://phabricator.wikimedia.org/T147432 for the Varnishkafka testing framework! [14:56:54] mforns: ok, we probably need to take a second look at those [14:57:31] elukey: if we feel strongly about that we can do it as part of operational excellence [14:58:12] nuria_: we can chat about it, the other varnish-event tool is not quite like varnishkafka [14:58:28] because it needs something that pushes events to kafka to work [14:58:43] elukey: ahaha [14:59:30] Analytics-Wikimetrics, Community-Wikimetrics: Inconsistent metrics results for 90 day rolling active editors calculation - https://phabricator.wikimedia.org/T147176#2693001 (Aklapper) Ah! Thanks a lot! All makes sense now, and sorry for my confusion! So this task is about #Community-Wikimetrics and #Anal... [15:06:06] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [30.0] [15:11:06] RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK: OK: Less than 20.00% above the threshold [20.0] [15:30:46] sorry about bad connection, I was saying don't we want ottomata around for goals? [15:34:02] a-team: we can meet now, not finding any better shops [15:34:24] milimetric, ok for me [15:40:19] nuria_: do you mind just dialing my phone in? The internet is against me today [15:40:31] milimetric: of course [15:41:32] milimetric: we talked with otto before as he knew he would be gone the 1st week of the quarter to get his feedback for public event stream and it is already there on teh tickets [15:44:26] milimetric: tried to call but going to voice mail [15:45:05] Analytics-Wikimetrics, Community-Wikimetrics: Inconsistent metrics results for 90 day rolling active editors calculation - https://phabricator.wikimedia.org/T147176#2693188 (gabrielle_marie_wmch) Ah, thank you so much. i am sorry i got the notion of the project wrong, I was thinking in terms of Wikimedia... [15:46:52] ping schana [15:47:42] Analytics-Kanban: Count pageviews for all wikis/systems behind varnish - https://phabricator.wikimedia.org/T130249#2693190 (Nuria) [15:54:17] wow, Whole Foods is my hero, amazing wifi, outlets, tons of space [15:54:26] I'm in business yall [15:54:58] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [30.0] [15:59:59] (PS1) Nuria: Bringing master up to date with aqs-new-cluster branch [analytics/aqs] - https://gerrit.wikimedia.org/r/314284 (https://phabricator.wikimedia.org/T144521) [16:00:41] milimetric: good, will note that [16:01:10] elukey: we need to merge our branch to master so we can deploy from master from now on, see CR: https://gerrit.wikimedia.org/r/#/c/314284/ [16:02:07] elukey: we need to merge this one before closing new AQS cluster ticket [16:02:48] (PS2) Nuria: Bringing master up to date with aqs-new-cluster branch [analytics/aqs] - https://gerrit.wikimedia.org/r/314284 (https://phabricator.wikimedia.org/T144497) [16:03:29] Analytics-Kanban, Datasets-Webstatscollector, RESTBase-Cassandra, Patch-For-Review: Better response times on AQS (Pageview API mostly) {melc} - https://phabricator.wikimedia.org/T124314#2693324 (Nuria) [16:03:41] nuria_: yes you are right! [16:04:17] Analytics-Kanban: Decomission hosts from old aqs cluster - https://phabricator.wikimedia.org/T147460#2693339 (Nuria) [16:05:09] Analytics-Kanban: Clean up references on puppet code to old AQS cluster - https://phabricator.wikimedia.org/T147461#2693355 (Nuria) [16:07:30] Analytics, Analytics-Kanban: Count pageviews for all wikis/systems behind varnish - https://phabricator.wikimedia.org/T130249#2693417 (Nuria) [16:07:46] you know, if what I said about the monitoring is right, then these Icinga alerts make no sense, they do seem to be triggered when there are invalid events... [16:07:47] so weird [16:08:24] Analytics-Kanban: Kill limn1 - https://phabricator.wikimedia.org/T146308#2693421 (Nuria) [16:11:07] Analytics: Capacity projections of pageview API document on wikitech - https://phabricator.wikimedia.org/T138318#2693438 (Nuria) Open>Resolved [16:12:07] Analytics-Dashiki, Analytics-Kanban: {crow} Dashiki Ops - https://phabricator.wikimedia.org/T102250#2693455 (Nuria) Open>Resolved [16:12:31] Analytics-Cluster, Continuous-Integration-Config: Jenkins tests for analytics/refinery? - https://phabricator.wikimedia.org/T147072#2693456 (madhuvishy) My patch is unrelated to this, and is specifically only for a step in the refinery deployment process. Not sure if the rest of the repo needs tests - Pi... [16:12:36] Analytics-Kanban, Patch-For-Review: Get jenkins to update refinery with deploy of new jars {hawk} - https://phabricator.wikimedia.org/T130123#2693458 (madhuvishy) @hashar Could you please merge https://gerrit.wikimedia.org/r/290640. The job is already deployed and has been used by Analytics for the past... [16:12:45] Analytics-Kanban, Analytics-Wikimetrics: {dove} Wikimetrics Ops - https://phabricator.wikimedia.org/T102227#2693462 (Nuria) Open>Resolved [16:13:04] Analytics-Dashiki, Analytics-Kanban: {frog} Limn -> Dashiki - https://phabricator.wikimedia.org/T88611#2693468 (Nuria) Open>Resolved [16:14:03] Analytics-Kanban: Scale MySQL edit history reconstruction data extraction - https://phabricator.wikimedia.org/T134791#2693488 (Nuria) Open>Resolved a:Nuria outdated, no longer relevant [16:14:05] Analytics-Kanban: Wikistats 2.0. Edit Reports: Setting up a pipeline to source Historical Edit Data into hdfs {lama} - https://phabricator.wikimedia.org/T130256#2693491 (Nuria) [16:15:33] Analytics: Figure out a rollback strategy if the release job fails {hawk} - https://phabricator.wikimedia.org/T132179#2693515 (madhuvishy) a:madhuvishy>None [16:15:51] Analytics: Send burrow lag statistics to statsd/graphite {hawk} - https://phabricator.wikimedia.org/T120852#2693517 (madhuvishy) a:madhuvishy>None [16:15:53] Analytics, Spike: Research spike: load enwiki data into Druid to study lookup table performance - https://phabricator.wikimedia.org/T141472#2693519 (Nuria) Isnt't this a duplicate of https://phabricator.wikimedia.org/T134792? [16:17:44] Analytics: Get rid of Pageview-API -> Analytics auto-tagging - https://phabricator.wikimedia.org/T146042#2693549 (Nuria) a:ggellerman [16:20:47] * elukey goes afk! [16:21:57] Analytics, Pageviews-API: Pageview API: Better filtering of bot traffic on top enpoints - https://phabricator.wikimedia.org/T123442#2693645 (Nuria) [16:21:59] Analytics: Top Pageview stats for August 27th doesn't look right - https://phabricator.wikimedia.org/T144715#2693647 (Nuria) [16:22:43] Analytics: Top Pageview stats for August 27th doesn't look right - https://phabricator.wikimedia.org/T144715#2608052 (Nuria) There is nothing special about this day that i can see but rather another instance of unfiltered bot traffic distorting top endpoints [16:22:56] Analytics: Top Pageview stats for August 27th doesn't look right - https://phabricator.wikimedia.org/T144715#2693656 (Nuria) Closing as duplicate [16:24:28] RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK: OK: Less than 20.00% above the threshold [20.0] [16:24:36] Analytics-Kanban, Operations, Performance-Team, Traffic: Preliminary Design document for A/B testing - https://phabricator.wikimedia.org/T143694#2693678 (Nuria) Working on doc https://docs.google.com/document/d/1jRGjVAthJXoCovxyvXWyg07R1POb8zvD_n8IlJXrPVM/edit# Will start addressing @ellery's la... [16:33:11] Analytics: Improve mediawiki data redaction and refactor edit history reconstruction - https://phabricator.wikimedia.org/T146444#2661250 (chasemp) Who does this mean? 'And then we need to refactor our history reconstruction on top of that' 'our' is analytics? We are in the midst of giving some time and at... [16:44:06] Analytics, Trash: --- Immediate Above ----------------------- - https://phabricator.wikimedia.org/T115634#2693817 (Nuria) [16:49:03] milimetric: the alarms are kind of puzzling true, but i think they might be working because we are adding events to the eventlogging_* topic (error topic) and thus ratios are off [16:49:08] milimetric: not in teh obvious way [16:49:29] milimetric: they are off cause there are MORE events in the eventlogging_* topics than they shoudl be [16:49:31] *should [16:52:17] Analytics-Kanban: Implement Pages Created & Count of Edits full vertical slice - https://phabricator.wikimedia.org/T131779#2693854 (Nuria) [16:54:04] Analytics, Analytics-Dashiki: Show pageviews prior to 2015 in dashiki - https://phabricator.wikimedia.org/T143906#2693870 (Nuria) [16:54:49] nuria_: hm... but eventlogging_* should include all events, valid or not. The fact that it is different from raw topic counts means events are getting lost somewhere. From what I see in the code, raw should equal eventlogging_* exactly unless there's a restart. [16:56:29] milimetric: or unless eventerror is not publishing as we think it is [16:58:16] Analytics-Kanban: Implement Pages Created & Count of Edits full vertical slice - https://phabricator.wikimedia.org/T131779#2693875 (Nuria) This task is been replaced by this one: https://phabricator.wikimedia.org/T143924 [16:58:25] Analytics-Kanban: Implement Pages Created & Count of Edits full vertical slice - https://phabricator.wikimedia.org/T131779#2693876 (Nuria) Closing as outdated [16:58:31] Analytics-Kanban: Wikistats 2.0. Edit Reports: Setting up a pipeline to source Historical Edit Data into hdfs {lama} - https://phabricator.wikimedia.org/T130256#2693878 (Nuria) [16:58:33] Analytics-Kanban: Implement Pages Created & Count of Edits full vertical slice - https://phabricator.wikimedia.org/T131779#2693877 (Nuria) Open>Resolved [16:58:37] nuria_: yeah, it could be there's some magic somewhere that keeps its counts from graphite? [16:59:04] anyway, when Andrew comes back I think he knows how that was set up, so I can change the code any which way when we understand what's going on [16:59:17] I could definitely do it now too, but seems like I'd waste a bunch of time [16:59:29] milimetric:we can also look at distinct counts on graphite.wikimedia.org [16:59:34] milimetric: let me see those [16:59:45] k, it'll be eventlogging_EventError [17:00:33] Analytics-Kanban, MediaWiki-extensions-WikimediaEvents, Collab-Team-Q1-July-Sep-2016, Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016), and 2 others: EL alarms raw/validated 20160926 - https://phabricator.wikimedia.org/T146674#2693896 (Etonkovidova) @Millmetric - a good question. I'll ask... [17:00:43] Analytics: Find out what happens to the old rows in the revision table - https://phabricator.wikimedia.org/T142535#2538504 (Nuria) cc @Milimetric , is this still relevant? [17:01:59] Analytics: Find out what happens to the old rows in the revision table - https://phabricator.wikimedia.org/T142535#2693908 (Milimetric) yes. We only know part of why this happens, but I have some saved data to look over that should help us figure it out. [17:06:11] milimetric: teh eventerror schema is not on graphite [17:06:14] *the [17:06:45] oh... weird, I wonder where that config is. We should at least update the code comments with that [17:07:43] but nuria_ I think the better way would be to send that topic's stats to graphite and use it for monitoring instead, it would be much easier to understand [17:07:44] milimetric: the eventError topic must live elsewhere , do look at http://graphhite.wikimedia.org and follow teh path from kafka [17:07:47] *the [17:09:11] milimetric: so despite what it might seem , it is not being counted on the metrics, so the raw/valid alarms make sense, right? [17:11:49] Analytics-Kanban: Examine puppet code for Event Logging and make sure monitoring is using the best counts - https://phabricator.wikimedia.org/T147321#2688873 (Nuria) As far as i can see graphite does not have a metric for eventlogging_EventError under the kafka metrics. [17:11:49] nuria_: it's not on some brokers, but it is on kafka1014 [17:12:17] must mean that $kafka_brokers_graphite_wildcard doesn't include that broker [17:13:10] it's actually on 1020 and 1022 too [17:13:12] weird :) [17:14:53] Analytics-Kanban: Examine puppet code for Event Logging and make sure monitoring is using the best counts - https://phabricator.wikimedia.org/T147321#2693935 (Milimetric) Actually, even weirder, the kafka1014, kafka1020, and kafka1022 have eventlogging_EventError but not the other brokers. There's a wildcar... [17:15:56] Analytics-Kanban, MediaWiki-extensions-WikimediaEvents, Collab-Team-Q1-July-Sep-2016, Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016), and 2 others: EL alarms raw/validated 20160926 - https://phabricator.wikimedia.org/T146674#2693937 (Milimetric) Right, @Etonkovidova, that's fine and no r... [17:21:52] a-team: Hi, I apologize for having missed today's meeting on goals. I had in mlind it was ahem, now :( [17:22:10] I'll be there on time tomorrow [17:22:26] Analytics, Pageviews-API, WMDE-Analytics-Engineering, Wikidata: "egranary digital library system" UA should be listed as a spider - https://phabricator.wikimedia.org/T135164#2693956 (Nuria) This is again another instance of bot traffic that slips by, this UA might not be causing trouble now but t... [17:22:53] Analytics, Research-and-Data-Backlog: Improve bot identification at scale - https://phabricator.wikimedia.org/T138207#2693962 (Nuria) [17:22:55] Analytics, Pageviews-API, WMDE-Analytics-Engineering, Wikidata: "egranary digital library system" UA should be listed as a spider - https://phabricator.wikimedia.org/T135164#2693964 (Nuria) [17:29:03] nuria_: yes? [17:45:05] Analytics-Kanban, MediaWiki-extensions-WikimediaEvents, Collab-Team-Q1-July-Sep-2016, Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016), and 2 others: EL alarms raw/validated 20160926 - https://phabricator.wikimedia.org/T146674#2694082 (Etonkovidova) @Millmetric the fix for validation works... [17:47:38] Analytics-Kanban, EventBus, Services, Wikimedia-Stream, User-mobrovac: Public Event Streams - https://phabricator.wikimedia.org/T130651#2141764 (Tomayac) Sorry, chiming in quite late based on Andrew Otto's ping 13 days ago via email… Server-Sent Events (SSE) is broadly supported on desktop... [17:47:51] Analytics-Kanban, EventBus, Services, Wikimedia-Stream, User-mobrovac: Public Event Streams - https://phabricator.wikimedia.org/T130651#2694087 (Tomayac) Sorry, chiming in quite late based on Andrew Otto's ping 13 days ago via email… Server-Sent Events (SSE) is broadly supported on desktop... [17:49:14] Analytics-Kanban, EventBus, Services, Wikimedia-Stream, User-mobrovac: Public Event Streams - https://phabricator.wikimedia.org/T130651#2694089 (Tomayac) (Sorry, I am on a flaky train Wi-Fi, it seems my comment got double-posted despite one failure message, removed one of the two). [18:06:18] Analytics: Improve mediawiki data redaction and refactor edit history reconstruction - https://phabricator.wikimedia.org/T146444#2694155 (Milimetric) Hi @chasemp, I couldn't explain this with fewer words, sorry, let's chat in person if that's easier: * Analytics is exporting data from mediawiki databases to... [18:09:18] Analytics-Kanban, EventBus, Services, Wikimedia-Stream, User-mobrovac: Public Event Streams - https://phabricator.wikimedia.org/T130651#2694171 (Milimetric) Thanks very much, @Tomayac, appreciate the analysis. [18:10:21] Analytics-Kanban, EventBus, Services, Wikimedia-Stream, User-mobrovac: Public Event Streams - https://phabricator.wikimedia.org/T130651#2694176 (GWicke) >>! In T130651#2663303, @Milimetric wrote: > Therefore, to me, SSE is pointless. It has no real advantage to socket.io or even simple polli... [18:14:25] schana: did you verify you have access and that you can run oozie jobs? [18:21:13] Analytics-Kanban, MediaWiki-extensions-WikimediaEvents, Collab-Team-Q1-July-Sep-2016, Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016), and 2 others: EL alarms raw/validated 20160926 - https://phabricator.wikimedia.org/T146674#2694204 (Catrope) >>! In T146674#2693937, @Milimetric wrote: >... [18:30:32] Analytics-Kanban, MediaWiki-extensions-WikimediaEvents, Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016), Patch-For-Review, WMF-deploy-2016-10-04_(1.28.0-wmf.21): EL alarms raw/validated 20160926 - https://phabricator.wikimedia.org/T146674#2694253 (jmatazzoni) [18:46:10] Analytics-Kanban: Roll up Sessions metric to weekly granularity - https://phabricator.wikimedia.org/T147492#2694320 (Milimetric) [18:51:09] Analytics-Kanban: Roll up Sessions metric to weekly granularity - https://phabricator.wikimedia.org/T147492#2694357 (Milimetric) @mforns I ran https://gist.github.com/milimetric/85e66805f658a23832c8e35dcbc4fb1c on the sessions output, it's pretty fast but sadly doesn't compress it as much as I thought. It c... [19:10:09] Analytics-Kanban, Continuous-Integration-Config, Patch-For-Review: Get jenkins to update refinery with deploy of new jars {hawk} - https://phabricator.wikimedia.org/T130123#2694414 (hashar) [19:11:47] Analytics-Kanban, Continuous-Integration-Config, Patch-For-Review: Get jenkins to update refinery with deploy of new jars {hawk} - https://phabricator.wikimedia.org/T130123#2694433 (hashar) @madhuvishy I don't scale anymore :] Best way is to add #continuous-integration-config to the task and poke fo... [19:14:57] milimetric/nuria_ Can I close the jenkins tasks on Kanban? They are in Done but not closed [19:15:11] madhuvishy: hola! [19:15:29] madhuvishy: i was hoping that code would be merged and thus close them but please do [19:15:43] nuria_: yeah it just got merged [19:15:50] madhuvishy: ooohhhhhh [19:15:57] :) yeah, madhuvishy, if that code never gets merged meh... not on you [19:15:59] madhuvishy: super good, thank you [19:16:12] :) [19:16:15] yeah, finally [19:16:59] oh yay [19:21:16] Analytics-Kanban: Get jenkins to automate releases {hawk} - https://phabricator.wikimedia.org/T130122#2694530 (madhuvishy) [19:21:18] Analytics-Kanban, Continuous-Integration-Config, Patch-For-Review: Add JJB support for Jenkins Maven Release Plugin {hawk} - https://phabricator.wikimedia.org/T132175#2694529 (madhuvishy) Open>Resolved [19:21:53] Analytics-Kanban: Make deployment process to the cluster easier, more streamlined {hawk} - https://phabricator.wikimedia.org/T129253#2694533 (madhuvishy) [19:21:55] Analytics-Kanban, Continuous-Integration-Config, Patch-For-Review: Get jenkins to update refinery with deploy of new jars {hawk} - https://phabricator.wikimedia.org/T130123#2694531 (madhuvishy) Open>Resolved Thanks @hashar! I'll verify that. [19:22:13] Analytics-Kanban: Get jenkins to automate releases {hawk} - https://phabricator.wikimedia.org/T130122#2126478 (madhuvishy) [19:22:15] Analytics-Kanban, Patch-For-Review: Translate the analytics-release-test job to YAML config in integration/config {hawk} - https://phabricator.wikimedia.org/T132182#2694534 (madhuvishy) Open>Resolved [19:22:39] Analytics-Kanban: Make deployment process to the cluster easier, more streamlined {hawk} - https://phabricator.wikimedia.org/T129253#2099799 (madhuvishy) [19:22:41] Analytics-Kanban: Get jenkins to automate releases {hawk} - https://phabricator.wikimedia.org/T130122#2126478 (madhuvishy) Open>Resolved [19:25:57] Analytics-Kanban: Roll up Sessions metric to weekly granularity - https://phabricator.wikimedia.org/T147492#2694560 (mforns) @Milimetric Oh! I didn't think about that. So, different days do not have all session paths... And when they get aggregated, the rows change date but can not be aggregated because the... [19:26:40] Analytics-Kanban: Roll up Sessions metric to weekly granularity - https://phabricator.wikimedia.org/T147492#2694562 (mforns) @Milimetric Awesome script BTW :] [19:28:17] milimetric: we still need to deploy the semantic 2.0 update right? [19:29:59] yes [19:30:05] nuria_: right, I can do that [19:37:09] milimetric: ok, let me know if you need help [20:00:58] (PS4) Nuria: [WIP] Service Worker to cache locally AQS data [analytics/dashiki] - https://gerrit.wikimedia.org/r/302755 (https://phabricator.wikimedia.org/T138647) [20:04:29] nuria_: I figured out what takes so long in this build, I think. So I have bad internet here and it randomly fails to build with: [20:04:41] some font error [20:04:44] Error: Broken @import declaration of "https://fonts.googleapis.com/css?family=Lato:400,700,400italic,700italic&subset=latin" - timeout [20:04:45] let me guess [20:04:47] yea [20:04:48] :) [20:05:01] those are css importing fonts from google [20:05:14] nuria_: restbase deployed with new throttling limits [20:05:17] yeah, and apparently some genius decided it was a great idea to actually try and resolve that URL [20:05:21] Pchelolo: excittingggggg [20:05:33] * milimetric reaches through the internet and pokes gulp geniuses in the eye [20:05:40] milimetric: i saw this talk about the million tricks you need to do with fonts [20:06:05] milimetric: we can fix that though,do we use that font? [20:07:45] (PS5) Nuria: [WIP] Service Worker to cache locally AQS data [analytics/dashiki] - https://gerrit.wikimedia.org/r/302755 (https://phabricator.wikimedia.org/T138647) [20:08:58] nuria_: yeah, we use the fonts but we can disable the check I hope [20:09:06] otherwise I'll write my own damn css minifier [20:09:21] Pchelolo: our traffic now hardly ever goes beyond 50 reqs [20:09:37] Pchelolo: so we will just wait and see whether we ever hit throttling limits [20:10:07] nuria_: as far as semantic 2 vs. semantic 1, I think the size is similar because we're importing piece by piece each component we need [20:10:38] I want to componentize better, wrapping css and everything in each component [20:10:45] milimetric: the css is slightly bigger but is fine i do not think there is nothing we can do [20:10:59] I was starting to look at System.js to replace require.js, it seems better [20:11:54] {processImport: false} is promising for the other issue [20:17:31] milimetric: ya, processimport should be the default [20:20:33] milimetric: mmm... i do not think wrapping css in every component is worthy of effort given that we are always going to have a layout and all the build revolves arround a layout , you will need css somewhere that guarantees component css can play with each other when together in a layout [20:22:00] (PS1) Milimetric: Clean up gulp build and fab deploy [analytics/dashiki] - https://gerrit.wikimedia.org/r/314348 [20:22:33] ok, nuria_, deployed everything besides vital signs (should I do that?) [20:22:54] also, nuria_ I found some problems while deploying, fixed here: https://gerrit.wikimedia.org/r/314348 [20:23:09] milimetric: was vital signs layout updated too, I thought it was... [20:23:39] nuria_: yeah, it's the metrics-by-project layout, I can update it, it's just the weird one I have to push code [20:24:21] milimetric: are we talking analytics.wikimedia.org or other layout in labs? [20:25:03] nuria_: I agree about componentizing from a performance point of view, but from a complexity point of view, the layout shouldn't have to know what CSS all its components need, that defeats the purpose of bundling and componentizing in the first place [20:25:25] With webpack or System.js (I think) you should be able to do the same kind of css bundling we do now with require.js optimizer [20:25:33] though I agree it's not urgent [20:25:51] nuria_: analytics.wikimedia.org/dashboards/vital-signs [20:25:55] (CR) Nuria: [C: 2 V: 2] Clean up gulp build and fab deploy [analytics/dashiki] - https://gerrit.wikimedia.org/r/314348 (owner: Milimetric) [20:26:55] milimetric: wait, thsi domain: https://analytics.wikimedia.org/dashboards/vital-signs/#projects=all,eswiki,itwiki,enwiki,jawiki,dewiki,ruwiki,frwiki/metrics=Pageviews [20:27:07] yeah, same one I said, yep [20:28:14] milimetric: ok, deploying analytics always requires committing build code as we have no other way to deploy to prod, is that what you were saying? [20:29:15] yeah, I was saying that one's not done yet, doing now [20:29:34] k, done and pushed [20:29:40] nuria_: ^ [20:30:08] (so that'll be deployed whenever puppet pulls) [20:31:08] milimetric: and regarding components, you will always need 1 place where to synchronize that css is not appearing twice in your page (same file 2 times) that synchronization now is done in the layout, we make sure all css files appears only once. If you package the css per component you would need to make it external to the app (bower modules) to avoid the [20:31:08] duplication of loading it more than once as if component a uses 1.css and 2.css and component B uses 1.css and 3.css you will be bundling 1.css twice [20:32:07] milimetric: maybe that was a bit long for irc, we can talk about it tomorrow [20:32:09] nuria_: nah, System.js and webpack handle that for you, it's pretty cool :) It's the way C# and ASP.NET used to do it with backpack [20:32:40] milimetric: only if each css unit you ever use is a distinct file [20:33:08] milimetric: and thus a distinct http request [20:33:14] milimetric: then, yes, [20:33:16] nuria_: no, it builds and minifies, optimizing the same way require.js optimizes js [20:33:32] obviously you have to tell it what bundles you want [20:33:39] but it optimizes within the bundles [20:33:48] in our case if we want to make sure not to duplicate, we just make one bundle [20:34:02] or if we know some CSS is only needed by one component, we only include it there, like custom CSS [20:34:24] milimetric:which is what we have in the layout, 1 bundle. [20:34:55] yea, but the layout then needs to know about all the components [20:35:08] spreading it out makes only the components responsible for what they each need [20:36:13] I'm just excited 'cause I've been waiting for a proper web component system since like 10 years ago, and it looks like System.js might be it [22:21:34] Analytics-EventLogging: PHP Notice: Undefined index: REQUEST_TIME in extensions/EventLogging/includes/EventLogging.php on line 56 - https://phabricator.wikimedia.org/T70629#2695340 (Krinkle) Open>Invalid The `REQUEST_TIME` code that assumed HTTP context and was problematic when called from the comma...