[02:24:40] <wikibugs>	 Analytics, Easy: [REQUEST] Extract search queries from HTTP_REFERER field for a Wikibook - https://phabricator.wikimedia.org/T144714#2669554 (Tbayer) >>! In T144714#2629977, @Larsnooden wrote: > Google Search Console (webmaster tools) might be useful.  Which Wikimedia group would that fall under? >  > Ab...
[07:28:55] <ottomata>	 hey hoooo
[07:29:28] <ottomata>	 joal_: can we find some time today to talk about mw history project?
[07:29:41] <ottomata>	 want to make sure i have a good understanding of things to share here at ops offsite
[08:58:03] <grrrit-wm>	 (CR) MarcoAurelio: "Right. Gracias. Do we need to have this deployed before or after wiki creation, and who can merge it?" [analytics/refinery] - https://gerrit.wikimedia.org/r/312810 (https://phabricator.wikimedia.org/T146612) (owner: MarcoAurelio)
[09:20:42] <joal_>	 Hey ottomata n
[09:20:47] <joal_>	 for sure we can find time :)
[09:23:15] <ottomata>	 joal_:  maybe in 1 h?
[09:23:25] <joal_>	 ottomata: perfect
[09:23:53] <joal_>	 ottomata: I'll be online, I let you ping me (twice if I don't answer, sometimes scala has me not noticing)
[09:27:33] <ottomata>	 haah ok
[10:18:00] <ottomata>	 yerrghghhh, joal_ don't think i can get away
[10:18:14] <joal_>	 ottomata: no prob
[10:18:14] <ottomata>	 um, maybe around 15:45?
[10:18:30] <joal_>	 ottomata: sure :)
[10:18:34] <ottomata>	 ok cool
[10:18:39] <ottomata>	 there is a coffee break scheduled then :)
[10:18:45] <joal_>	 same as now, please ping ;)
[10:19:02] <joal_>	 ottomata: if you prefer one night when there would be quiet, feasible as well
[10:19:50] <ottomata>	 hm, ok maybe!
[10:19:55] <ottomata>	 let'st try 15:45 for now
[10:20:03] <joal_>	 k
[13:46:16] <ottomata>	 yoohoo
[13:46:16] <ottomata>	 joal_:
[13:46:18] <ottomata>	 :)
[13:46:49] <mforns>	 hi ottomata :]
[13:48:39] <ottomata>	 mforns:  hiyaaa
[13:48:44] <ottomata>	 yall wanna hang  out w me real quick?
[13:49:12] <ottomata>	 i am in batcave
[13:49:18] <mforns>	 ottomata, sure
[13:49:19] <ottomata>	 and/or milimetric:   too?
[13:59:34] <wikibugs>	 Analytics, Research-and-Data, Research-collaborations, Research-management, Patch-For-Review: Oozie job to extract data for WDQS research - https://phabricator.wikimedia.org/T146064#2670634 (leila) @Nuria , re X-Analytics header: I'd like for us to keep the this field in the extract for now....
[14:22:07] <joal_>	 ottomata: !!!
[14:22:40] <joal_>	 ottomata: man I'm very sorry
[14:22:57] <joal_>	 mforns: I assume reading the chan that you've spent some time with ottomata
[14:23:10] <mforns>	 joal_, hi! we're still in the batcave
[14:23:16] <joal_>	 joining !
[14:23:17] <mforns>	 join :]
[14:48:18] <grrrit-wm1>	 (CR) Joal: "Review needed for style and correctness." (6 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/301837 (https://phabricator.wikimedia.org/T141548) (owner: Joal)
[14:49:55] <ottomata>	 joal_:  i forgot to ping you twice!!! :p
[14:49:59] <ottomata>	 no probs!
[14:50:28] <joal_>	 ottomata: I actually forgot :( I was happy having a working patch, so launched a long job and went take a break :(
[14:50:40] <joal_>	 ottomata: I apologize again !
[14:50:52] <joal_>	 ottomata: it's good mforns and milimetric were here :)
[14:51:54] <ottomata>	 np ja i got a lot of goodies to work with
[14:52:08] <joal_>	 that's great ottomata :)
[15:09:05] <wikibugs>	 Analytics, Analytics-Dashiki: Add external link to tabs layout - https://phabricator.wikimedia.org/T146774#2670812 (Milimetric)
[15:10:10] <wikibugs>	 Analytics-Kanban: Make and deploy simple proof of concept dashboard for Daily Edits and Daily Pages Created on simplewiki - https://phabricator.wikimedia.org/T146775#2670825 (Milimetric)
[15:15:00] <milimetric>	 nuria_: how do I deploy to analytics.wikimedia.org again, is that written on a wiki somewhere?
[15:15:08] <milimetric>	 sorry I keep forgetting
[15:15:45] <nuria_>	 milimetric: yes, https://github.com/wikimedia/analytics.wikimedia.org/blob/master/README.md
[15:15:50] <milimetric>	 thx!
[15:20:29] <wikibugs>	 Analytics-Wikistats: Wikistats summary for Report Card has English/Russian Wikipedia-Zero traffic as separate entries instead of added to English/Russian Wikipedia mobile traffic - https://phabricator.wikimedia.org/T146777#2670874 (ezachte)
[15:22:36] <grrrit-wm1>	 (PS1) Milimetric: Add standard metrics dashboard [analytics/analytics.wikimedia.org] - https://gerrit.wikimedia.org/r/313024 (https://phabricator.wikimedia.org/T146775)
[15:23:01] <grrrit-wm1>	 (CR) Milimetric: [C: 2 V: 2] Add standard metrics dashboard [analytics/analytics.wikimedia.org] - https://gerrit.wikimedia.org/r/313024 (https://phabricator.wikimedia.org/T146775) (owner: Milimetric)
[15:25:50] <milimetric>	 nuria_: and does puppet get latest of that or do we have to login somewhere and pull?
[15:26:00] <nuria_>	 milimetric: puppet will push
[15:26:04] <milimetric>	 k
[15:26:08] <milimetric>	 what instance is it on?
[15:26:42] <nuria_>	 i do not remember, but does it matter ? as you do not need to ssh
[15:26:51] <milimetric>	 no, just curious, np
[15:27:04] <milimetric>	 mforns: how the queries going?
[15:27:34] <mforns>	 milimetric, mmm I was not working on them, was doing pivot load test instead...
[15:27:43] <mforns>	 should I prioritize the queries?
[15:27:50] <milimetric>	 oh, np, I'll run them quickly
[15:31:23] <nuria_>	 joal_: retroooo
[15:31:26] <joal_>	 arf
[15:59:03] <milimetric>	 joal_: where do you have the latest data for simplewiki, is it a table?
[15:59:15] <mforns>	 nuria_, you look jealous of being in barcelona's beach :]
[15:59:23] <nuria_>	 mforns: YES!
[15:59:26] <milimetric>	 haha
[15:59:26] <joal_>	 milimetric: I have finnaly found my bug for page not being user populated
[15:59:31] <nuria_>	 mforns: i want teletransport NOW
[15:59:33] <joal_>	 :)
[15:59:45] <mforns>	 nuria_, you mentioned it like 5 times :D
[16:00:01] <joal_>	 milimetric: I'll have that data in a parquet file queriable through spark or hive soon
[16:00:01] <milimetric>	 joal_: that's great, but I don't need anything perfect, just the latest so I can run these queries and put up these dashboards
[16:00:05] <nuria_>	 mforns: i know.....
[16:00:06] <milimetric>	 oh ok
[16:00:10] <nuria_>	 mforns: sadness
[16:00:13] <mforns>	 hehe
[16:00:16] <grrrit-wm1>	 (CR) Nuria: "Code is merged, there is nothing additional to do before wiki is created. It can be created now." [analytics/refinery] - https://gerrit.wikimedia.org/r/312810 (https://phabricator.wikimedia.org/T146612) (owner: MarcoAurelio)
[16:00:19] <milimetric>	 cool joal_, lemme know, whatever you have by today and I'll review resumes until then
[16:00:20] <joal_>	 milimetric: on it, will be done in 1/2 hour max
[16:00:36] <nuria_>	 mforns, milimetric , joal: BTW, there is a new wiki in the way: https://gerrit.wikimedia.org/r/#/c/312810/
[16:00:42] <milimetric>	 perfect, gives me time to eat and write the queries
[16:00:49] <joal_>	 nuria_: Yes, saw that
[16:01:10] <mforns>	 nuria_, OK
[16:01:14] <joal_>	 they did their homework very well, patch for whitelist before any alarm, that's good :)
[16:01:23] <joal_>	 nuria_: --^
[16:01:48] <nuria_>	 joal_: We decomented it on the "things to do before launching a wiki" list
[16:02:02] <joal_>	 nuria_: sure, that's good :)
[16:13:31] <wikibugs>	 Analytics-Kanban, Patch-For-Review: Switch AQS to new cluster - https://phabricator.wikimedia.org/T144497#2670984 (Nuria) {F4530091}
[16:16:30] <joal_>	 milimetric: data is ready here: /user/joal/mwhist_3/denorm_2
[16:17:22] <joal_>	 milimetric: user is populated, but there still is some bizzare things around is_reverted --> I got different results every time I run the denorm (not very satisfying)
[16:18:16] <milimetric>	 woohoo
[16:18:31] <milimetric>	 and the new schema has some new fields, right?
[16:18:37] <milimetric>	 you got a list of the fields?
[16:18:39] <joal_>	 milimetric: correct !
[16:18:53] <joal_>	 milimetric: will provide, give me a minute
[16:18:54] <milimetric>	 (was gonna make a hive table)
[16:18:56] <milimetric>	 thx
[16:19:20] <milimetric>	 hm, different results on denorm, interesting
[16:20:31] <joal_>	 milimetric: https://gist.github.com/jobar/fca828d5f6613cc49cbe2107652a3337
[16:20:45] <joal_>	 milimetric: very weird
[16:21:03] <nuria_>	 milimetric: did you deploy analytics.wikimedia?
[16:21:20] <joal_>	 milimetric: I assume it comes from the new strategy I use for perf optimisation
[16:25:05] <milimetric>	 nuria_: I pushed the code, the new dashboard is empty at: https://analytics.wikimedia.org/dashboards/standard-metrics/
[16:25:34] <milimetric>	 joal_: k, we'll help you brainbounce later if you want.  For now I do this and  get lunch :)
[16:49:56] <wikibugs>	 Analytics-Kanban, Patch-For-Review: Make and deploy simple proof of concept dashboard for Daily Edits and Daily Pages Created on simplewiki - https://phabricator.wikimedia.org/T146775#2670825 (Milimetric) Dashboard up at https://analytics.wikimedia.org/dashboards/standard-metrics/
[16:49:59] <milimetric>	 all right, nuria_, dashboard's populated: https://analytics.wikimedia.org/dashboards/standard-metrics/#simplewiki
[16:49:59] <milimetric>	 cc mforns_ / joal_ ^ that's our vertical slice
[17:15:13] <mforns>	 nuria_, yt?
[17:24:38] <nuria_>	 mforns: on interview back in abit
[17:38:55] <nuria_>	 mforns_: back, what's up?
[17:39:13] <mforns_>	 hi nuria_, about the pivot load test, can we batcave?
[17:39:34] <nuria_>	 mforns_: yes, give me 10 mins ok? :50 after the hour?
[17:39:38] <icinga-wm_>	 PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 26.67% of data above the critical threshold [30.0]
[17:39:43] <mforns_>	 nuria_, sure
[17:40:00] <mforns_>	 oh, there you have the alerts we were saying
[17:43:44] <nuria_>	 mforns_: right, the code is not deployed yet i do not think, i will be so this week
[17:48:59] <nuria_>	 mforns_: ready when you are
[17:49:07] <mforns_>	 nuria_, omw!
[17:49:35] <icinga-wm_>	 RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK: OK: Less than 20.00% above the threshold [20.0]
[17:51:08] <joal>	 Dropping for today a-team
[17:51:15] <joal>	 see you tomorrow
[18:09:17] <mforns_>	 bye joal!
[18:56:05] <wikibugs>	 Analytics-Kanban: Special characters showing up as question marks in /pageviews/top endpoint - https://phabricator.wikimedia.org/T145043#2671596 (Nuria) Please note that a bugfix going into mediawiki next week is going to deal with many requests that are now returning 200 when they should return 404: https:/...
[19:06:23] <wikibugs>	 Analytics-Kanban: Special characters showing up as question marks in /pageviews/top endpoint - https://phabricator.wikimedia.org/T145043#2671633 (Nuria) >Filter from top list the pages having a ratio (# views / # distinct user_agents) too high: This might run into false positives, for example  "trending page...
[19:13:16] <wikibugs>	 Analytics, Fundraising-Analysis, Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Provide performant query access to banner show/hide numbers - https://phabricator.wikimedia.org/T90649#2671656 (Nuria) This can be solved in two ways:  1. Process webrequest records every hour to a table tha...
[19:13:34] <nuria_>	 milimetric: i think this is another use case for streaming
[19:13:37] <nuria_>	 milimetric: https://phabricator.wikimedia.org/T90649
[20:24:48] <icinga-wm_>	 PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [30.0]
[20:27:21] <icinga-wm_>	 RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK: OK: Less than 20.00% above the threshold [20.0]
[20:33:19] <wikibugs>	 Analytics, MediaWiki-extensions-WikimediaEvents, The-Wikipedia-Library, Wikimedia-General-or-Unknown, Patch-For-Review: Implement Schema:ExternalLinksChange - https://phabricator.wikimedia.org/T115119#2671858 (Legoktm) >>! In T115119#2644117, @kaldari wrote: > If someone adds a link to a popu...
[20:50:29] <wikibugs>	 Analytics-EventLogging: Some recent ExternalLinksChange data lost - https://phabricator.wikimedia.org/T146815#2671929 (Samwalton9)
[20:52:56] <wikibugs>	 Analytics, Analytics-EventLogging: Some recent ExternalLinksChange data lost - https://phabricator.wikimedia.org/T146815#2671950 (Milimetric)