[02:04:00] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10TechCom-RFC: RFC: Modern Event Platform - Choose Schema Tech - https://phabricator.wikimedia.org/T198256 (10demon) Have you looked into protocol buffers? [02:54:12] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10TechCom-RFC: RFC: Modern Event Platform - Choose Schema Tech - https://phabricator.wikimedia.org/T198256 (10Ottomata) Yeah, both protobufs and thrift are options, but neither have the advantages that Avro does, but many of the same disadvantages. [05:04:09] 10Analytics, 10Operations, 10ops-eqiad, 10User-Elukey: Degraded RAID on dbstore1002 - https://phabricator.wikimedia.org/T197707 (10Marostegui) Still on-going ``` root@dbstore1002:~# megacli -PDRbld -ShowProg -PhysDrv [32:5] -aALL Rebuild Progress on Device at Enclosure 32, Slot 5 Completed 44% in 1019 Min... [06:16:00] hello people, as FYI on notebook100[3,4] and stat100[5,6] there were load issues and too many open files due to the recent labstore1006 outage [07:47:03] (03PS1) 10Addshore: Send maxlag & dispatch lag directly to graphite [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/443036 [07:47:13] (03CR) 10jerkins-bot: [V: 04-1] Send maxlag & dispatch lag directly to graphite [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/443036 (owner: 10Addshore) [07:47:42] (03CR) 10Addshore: "recheck" [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/443036 (owner: 10Addshore) [08:18:48] 10Analytics, 10Operations, 10Traffic: Size of headers processed by varnish? - https://phabricator.wikimedia.org/T198152 (10ema) p:05Triage>03Normal [08:27:09] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Varnishkafka eventlogging instances delivery failures - https://phabricator.wikimedia.org/T198070 (10elukey) In the case outlined in the description, there seemed to be a problem with kafka-jumbo1005 reading from librdkafka logs. I checked... [08:39:26] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Varnishkafka eventlogging instances delivery failures - https://phabricator.wikimedia.org/T198070 (10elukey) From https://github.com/confluentinc/confluent-kafka-python/issues/284, that seems more or less the same issue, I went to https://... [08:51:08] 10Analytics, 10Operations, 10Traffic: Size of headers processed by varnish? - https://phabricator.wikimedia.org/T198152 (10ema) Both [[ https://varnish-cache.org/docs/5.1/reference/varnishd.html#http-req-hdr-len | varnish ]] and [[http://nginx.org/en/docs/http/ngx_http_core_module.html#large_client_header_bu... [08:51:24] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Varnishkafka eventlogging instances delivery failures - https://phabricator.wikimedia.org/T198070 (10elukey) Very nice info about broker issues and how librdkafka deals with them: https://github.com/edenhill/librdkafka/wiki/Producer-messa... [09:00:41] very interesting things about varnishkafka --^ [09:01:19] I also refactored again the dashboard :) https://grafana.wikimedia.org/dashboard/db/varnishkafka?orgId=1 [09:02:39] (03CR) 10Jonas Kress (WMDE): [C: 032] Send maxlag & dispatch lag directly to graphite [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/443036 (owner: 10Addshore) [09:03:03] (03Merged) 10jenkins-bot: Send maxlag & dispatch lag directly to graphite [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/443036 (owner: 10Addshore) [09:03:06] (03PS1) 10Addshore: Send maxlag & dispatch lag directly to graphite [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/443053 [09:03:09] (03CR) 10Addshore: [C: 032] Send maxlag & dispatch lag directly to graphite [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/443053 (owner: 10Addshore) [09:03:12] (03Merged) 10jenkins-bot: Send maxlag & dispatch lag directly to graphite [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/443053 (owner: 10Addshore) [09:06:09] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10TechCom-RFC: RFC: Modern Event Platform - Choose Schema Tech - https://phabricator.wikimedia.org/T198256 (10phuedx) >>! In T198256#4323745, @Ottomata wrote: > Yeah, both protobufs and thrift are options, but neither have the advantages that Avro does, ye... [09:09:55] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10TechCom-RFC: RFC: Modern Event Platform - Choose Schema Tech - https://phabricator.wikimedia.org/T198256 (10phuedx) > Full JSONSchema spec is too featureful and extensive for easy integration between different systems. Could you go into more detail on t... [09:52:04] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Intervals/buckets for data arround pageviews per country in wikistats maps - https://phabricator.wikimedia.org/T188928 (10fdans) moving back to in progress as there's still the patch for the UI pending, thank you @JAllemandou :) [10:26:22] * elukey lunch + errand! [10:30:57] (03PS1) 10Jonas Kress (WMDE): Track number of editors from other Wikimedia projects who also edit on Wikidata over time [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/443069 (https://phabricator.wikimedia.org/T193641) [11:21:30] (03CR) 10Addshore: Track number of editors from other Wikimedia projects who also edit on Wikidata over time (033 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/443069 (https://phabricator.wikimedia.org/T193641) (owner: 10Jonas Kress (WMDE)) [11:26:24] (03CR) 10Jonas Kress (WMDE): Track number of editors from other Wikimedia projects who also edit on Wikidata over time (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/443069 (https://phabricator.wikimedia.org/T193641) (owner: 10Jonas Kress (WMDE)) [11:27:25] joal: now this is a whole other thing now https://usercontent.irccloud-cdn.com/file/xG9MRLO9/Screen%20Shot%202018-06-29%20at%201.26.50%20PM.png [11:36:48] Wow fdans ! [11:37:17] fdans: I don't understand the color coding super well, but indeed I can see differences here :) [11:37:35] joal: a granularity of 1000 and a log scale adds a lot more depth to the map [11:37:47] the closer to yellow the higher [11:37:58] (needs a legend, which is what I'm doing now) [11:39:52] fdans: In term of visual, I think I understand better "The darker the higher", but it's personal matter I guess [11:40:28] joal: that makes sense, especially considering that countries below 100 are reported as white [11:40:45] true [11:41:55] joal yep, this might be better (but less rockandroll) https://usercontent.irccloud-cdn.com/file/Kx7CdOSM/Screen%20Shot%202018-06-29%20at%201.41.13%20PM.png [11:42:55] fdans: Makes it easier to read for me indeed - But a "Rock-N_Roll" setting for interesting color-scales would definitely be an add-on we should onsider [11:43:34] This ceiled value change was a super good call :) [11:43:38] Thanks a lot fdans :) [11:44:38] joal: that was a great team decision :) [11:53:25] (03CR) 10Joal: "Comments inline - Not far from good to merge :)" (035 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/443069 (https://phabricator.wikimedia.org/T193641) (owner: 10Jonas Kress (WMDE)) [12:11:31] (03CR) 10Addshore: Track number of editors from other Wikimedia projects who also edit on Wikidata over time (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/443069 (https://phabricator.wikimedia.org/T193641) (owner: 10Jonas Kress (WMDE)) [13:01:36] 10Analytics, 10Analytics-Kanban, 10Operations, 10User-Elukey: Tune Varnishkafka delivery errors to be more sensitive - https://phabricator.wikimedia.org/T173492 (10elukey) [13:24:40] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10TechCom-RFC: RFC: Modern Event Platform - Choose Schema Tech - https://phabricator.wikimedia.org/T198256 (10Ottomata) > AFAICT JavaScript, PHP, Python, and Java all have libraries for JSONSchema draft 6. The languages do yes, but we want to easily impor... [13:26:19] i have a dentist apointment this morning, so i'll be out for a bit, but should make it back before standup [13:31:36] ack! [13:55:31] 10Analytics, 10User-Elukey: Upgrade AQS to Debian Stretch - https://phabricator.wikimedia.org/T196138 (10elukey) Updating after a chat with Eric. In theory, if we manage to keep the current Raid 10 config on each node and format only the root partition, when the instance boots up we should be able to just star... [13:55:42] joal: o/ [13:55:44] about --^ [13:56:03] any thought/concern? [14:33:52] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10TechCom-RFC: RFC: Modern Event Platform - Choose Schema Tech - https://phabricator.wikimedia.org/T198256 (10phuedx) >>! In T198256#4324763, @Ottomata wrote: > … we need schemas to be predicable, types to be static, and field names to be SQL compatible (n... [14:35:27] nuria_: around? [14:35:29] ping [14:37:38] (03CR) 10Jonas Kress (WMDE): Track number of editors from other Wikimedia projects who also edit on Wikidata over time (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/443069 (https://phabricator.wikimedia.org/T193641) (owner: 10Jonas Kress (WMDE)) [15:20:20] 10Analytics, 10Discovery-Search (Current work): Use kafka for communication from analytics cluster to elasticsearch - https://phabricator.wikimedia.org/T198490 (10Gehel) [15:24:26] 10Analytics, 10Discovery-Search (Current work): Use kafka for communication from analytics cluster to elasticsearch - https://phabricator.wikimedia.org/T198490 (10Ottomata) I am curious, what would the data look like? Would it be possible to use something like https://github.com/confluentinc/kafka-connect-ela... [15:26:42] (03PS7) 10Sahil505: Added pagecounts metric [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442257 (https://phabricator.wikimedia.org/T189619) [15:38:52] (03CR) 10Fdans: "Sahil, thank you for doing this change! It'll provide Wikistats 2 with a lot more historical depth. I've just a few comments about naming " (036 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442257 (https://phabricator.wikimedia.org/T189619) (owner: 10Sahil505) [15:41:53] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 3 others: [EPIC] Develop a JobQueue backend based on EventBus - https://phabricator.wikimedia.org/T157088 (10mobrovac) [15:41:58] 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10Goal, and 2 others: FY17/18 Q4 Program 8 Services Goal: Complete the JobQueue transition to EventBus - https://phabricator.wikimedia.org/T190327 (10mobrovac) 05Open>03Resolved a:03Pchelolo We are done, mesdames et messieurs. \o/ \o/ \o/ [15:48:16] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10TechCom-RFC, 10Services (designing): RFC: Modern Event Platform - Choose Schema Tech - https://phabricator.wikimedia.org/T198256 (10mobrovac) [15:54:18] (03PS8) 10Mforns: Added pagecounts metric [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442257 (https://phabricator.wikimedia.org/T189619) (owner: 10Sahil505) [15:56:05] elukey: Was gone for kids - Sounds good to me (the cassandra plan :) Monday? [15:59:08] yes! :) [15:59:35] great :) [16:00:28] ping mforns fdans milimetric [16:06:17] YARGH [16:06:19] internet being flaky [16:06:20] v sorry [16:06:22] trying to join [16:23:33] (03PS9) 10Sahil505: Added pagecounts metric [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442257 (https://phabricator.wikimedia.org/T189619) [16:24:13] 10Analytics, 10Discovery-Search (Current work): Use kafka for communication from analytics cluster to elasticsearch - https://phabricator.wikimedia.org/T198490 (10EBernhardson) I don't know if we could use a generic endpoint, although we could look into it. These updates are somewhat unique in that we send a... [16:33:41] (03CR) 10Sahil505: [C: 031] "Thanks a lot Marcel for constant help in finishing this :-]" (035 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442257 (https://phabricator.wikimedia.org/T189619) (owner: 10Sahil505) [16:46:46] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Update piwik to latest stable - https://phabricator.wikimedia.org/T192298 (10elukey) Going back to in progress, some issues are coming up after the upgrade: 1) Nuria had to restart Matomo (apachectl restart) due to slowness and timeouts.... [16:46:57] nuria_: --^ [16:50:01] 10Analytics, 10Discovery-Search (Current work): Use kafka for communication from analytics cluster to elasticsearch - https://phabricator.wikimedia.org/T198490 (10Ottomata) wait for kubernetes and stream processing? :D [16:50:43] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Update piwik to latest stable - https://phabricator.wikimedia.org/T192298 (10Nuria) I bet that error comes from me using a session in piwik that predated the upgrade [17:00:32] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Update piwik to latest stable - https://phabricator.wikimedia.org/T192298 (10elukey) It happens even now, and httpd returns `application/x-httpd-php/400`. The requests are look the same, GET with long query strings.. (not all the same clie... [17:00:40] * elukey off!! [17:14:34] (03CR) 10Nuria: Added pagecounts metric (032 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442257 (https://phabricator.wikimedia.org/T189619) (owner: 10Sahil505) [17:30:45] 10Analytics, 10Patch-For-Review, 10Wikipedia-Android-App-Backlog (Android-app-release-v2.7.24x-I-Ice-lolly): Include HTTP Referer header when navigating through internal links - https://phabricator.wikimedia.org/T192779 (10Tbayer) Great, thanks! From a quick look at the data, this seems to have rolled out ar... [17:50:35] (03CR) 10Nuria: "Meta comment:" (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/443069 (https://phabricator.wikimedia.org/T193641) (owner: 10Jonas Kress (WMDE)) [18:11:27] 10Analytics, 10Operations, 10Traffic: Size of headers processed by varnish? - https://phabricator.wikimedia.org/T198152 (10Nuria) Ya, 8k seems quite a bit, not sure why would we need more than that in either end. [18:48:25] (03PS10) 10Mforns: Added pagecounts metric [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442257 (https://phabricator.wikimedia.org/T189619) (owner: 10Sahil505) [18:49:57] (03CR) 10Mforns: [V: 032 C: 031] "Looks great :]" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442257 (https://phabricator.wikimedia.org/T189619) (owner: 10Sahil505) [18:57:52] a-team, I deployed Wikistats2::pagecounts to the canary, feel free to review. I think the code is ready to go. https://wikistats-canary.wmflabs.org/pagecounts https://gerrit.wikimedia.org/r/#/c/analytics/wikistats2/+/442257/ [19:07:39] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Disable filter & split radio switches for metrics & reduce the height of wiki div in sidebar of detail page - https://phabricator.wikimedia.org/T198183 (10sahil505) [19:07:43] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Correct order of metric types on Dashboard navbar/menu - https://phabricator.wikimedia.org/T198168 (10sahil505) [19:07:46] 10Analytics-Kanban, 10Google-Summer-of-Code (2018): [Analytics] Improvements to Wikistats2 front-end - https://phabricator.wikimedia.org/T189210 (10sahil505) [19:07:48] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Correct spelling mistake in description meta tag of wikistats 2 - https://phabricator.wikimedia.org/T198122 (10sahil505) [19:17:51] 10Analytics: [Wikistats2] Negative numbers in chart axis are not abbreviated - https://phabricator.wikimedia.org/T198510 (10mforns) [19:18:49] 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10Services (later): Enable EventBus on all wikis - https://phabricator.wikimedia.org/T185170 (10Ottomata) Can this be closed? [19:21:04] 10Analytics, 10EventBus, 10Services (later): Reliable (atomic) MediaWiki event production - https://phabricator.wikimedia.org/T120242 (10Ottomata) [19:35:37] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Services (watching): Modern Event Platform (with EventLogging of the Future (EoF)) - https://phabricator.wikimedia.org/T185233 (10Ottomata) Ok! Q4 is over, and we've completed the interview process. Marshall, Dan and I spoke with... [19:38:17] (03CR) 10Nuria: "Thank you, visually is a lot better now with options disabled in selector. Please take a look at splits by "access methods" as I think the" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442257 (https://phabricator.wikimedia.org/T189619) (owner: 10Sahil505) [19:49:42] (03CR) 10Nuria: [C: 04-1] "How to repro bug:" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442257 (https://phabricator.wikimedia.org/T189619) (owner: 10Sahil505) [20:10:52] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Services (watching): Modern Event Platform (with EventLogging of the Future (EoF)) - https://phabricator.wikimedia.org/T185233 (10Ottomata) [20:24:52] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Partially purge MobileWikiAppiOSUserHistory eventlogging schema - https://phabricator.wikimedia.org/T195269 (10mforns) Hi @chelsyx! Also forgive me for the late response, your thorough report gave me a lot to think about. > However,... [21:37:03] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Services (watching): Modern Event Platform (with EventLogging of the Future (EoF)) - https://phabricator.wikimedia.org/T185233 (10Neil_P._Quinn_WMF) Thanks for the great summary, @Ottomata! >>! In T185233#4326260, @Ottomata wrote:...