[01:39:31] 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10Notifications, and 2 others: Make EchoNotification job JSON-serializable - https://phabricator.wikimedia.org/T192945#4177156 (10jmatazzoni) [02:18:02] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Wikimedia-Logstash, and 2 others: EventBus HTTP Proxy service does not report errors to logstash - https://phabricator.wikimedia.org/T193230#4177174 (10Pchelolo) @Ottomatta so we are still not getting proper logs right? At least I can't find them ;( Should we... [04:37:42] 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10MediaWiki-extensions-Translate, 10Services (done): Unable to mark pages for translation in Meta - https://phabricator.wikimedia.org/T192107#4177471 (10Etonkovidova) [04:37:47] 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10Goal, and 3 others: FY17/18 Q4 Program 8 Services Goal: Complete the JobQueue transition to EventBus - https://phabricator.wikimedia.org/T190327#4177472 (10Etonkovidova) [05:59:41] o/ [05:59:41] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Reimage the Debian Jessie Analytics worker nodes to Stretch. - https://phabricator.wikimedia.org/T192557#4177512 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts: ``` ['analytics1039.eqiad.wmnet', 'an... [05:59:47] reimaging two worker nodes --^ [06:23:25] 10Analytics, 10Analytics-Kanban: Upgrade Druid clusters to 0.11 - https://phabricator.wikimedia.org/T193712#4177523 (10elukey) p:05Triage>03High [06:44:41] Hi elukey [06:46:03] o/ [06:46:32] elukey: anything you'd like me to help with? [06:47:04] nope thanks :) [06:47:16] ok :) [06:48:26] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Reimage the Debian Jessie Analytics worker nodes to Stretch. - https://phabricator.wikimedia.org/T192557#4177558 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['analytics1039.eqiad.wmnet', 'analytics1040.eqiad.wmnet'] ``` and were **ALL** su... [06:57:53] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Upgrade Druid clusters to 0.11 - https://phabricator.wikimedia.org/T193712#4177574 (10elukey) [08:33:16] 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10MediaWiki-extensions-Translate, 10Services (done): Unable to mark pages for translation in Meta - https://phabricator.wikimedia.org/T192107#4177722 (10mobrovac) [08:33:22] 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10Goal, and 3 others: FY17/18 Q4 Program 8 Services Goal: Complete the JobQueue transition to EventBus - https://phabricator.wikimedia.org/T190327#4177723 (10mobrovac) [09:15:46] joal: cassandra news for us [09:16:08] I managed to rename our metrics to the 3.x format so now we share these dashboards with services [09:16:35] https://grafana.wikimedia.org/dashboard/db/cassandra [09:16:50] https://grafana.wikimedia.org/dashboard/db/cassandra-client-request?orgId=1 [09:16:57] https://grafana.wikimedia.org/dashboard/db/cassandra-read-repair?orgId=1 [09:17:03] https://grafana.wikimedia.org/dashboard/db/cassandra-system?orgId=1 [09:17:09] https://grafana.wikimedia.org/dashboard/db/cassandra-tables?orgId=1 [09:17:15] https://grafana.wikimedia.org/dashboard/db/cassandra-threadpools?orgId=1 [09:17:36] the trick though is to change "datasource" in the upper left corner to analytics-eqiad beforehand [09:18:43] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Unify, if possible, AQS and Restbase's cassandra dashboards - https://phabricator.wikimedia.org/T193017#4177834 (10elukey) Relabeling/naming done! These dashboards are now shared between Services and Analytics: https://grafana.wikimedia.o... [09:19:01] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Unify, if possible, AQS and Restbase's cassandra dashboards - https://phabricator.wikimedia.org/T193017#4177837 (10elukey) [09:21:56] ack elukey ! [09:22:15] (those dashboards are awesome btw) [10:21:57] going afk for early lunch + errand! [11:37:41] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4178076 (10Psychoslave) [11:45:17] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4178090 (10Psychoslave) As a first comment, please let me make clear that I'm open to any change on the ticket description if it is judged to partial. The goal is to resolve the issue s... [12:05:14] 10Analytics, 10Product-Analytics, 10Reading List Service, 10Reading-Infrastructure-Team-Backlog, and 3 others: [EPIC] Reading List Sync service analytics - https://phabricator.wikimedia.org/T191859#4178117 (10Tbayer) >>! In T191859#4158432, @mpopov wrote: >>>! In T191859#4156112, @Tbayer wrote: >>> Method... [12:26:11] 10Analytics, 10Product-Analytics, 10Reading List Service, 10Reading-Infrastructure-Team-Backlog, and 3 others: [EPIC] Reading List Sync service analytics - https://phabricator.wikimedia.org/T191859#4178146 (10Tbayer) > The reason to hash `app_install_id` is because these events would end up somewhere where... [13:39:58] helooo [14:04:08] 10Analytics, 10Operations: dbstore1002 disk 5 not healthy - https://phabricator.wikimedia.org/T193738#4178362 (10Marostegui) [14:08:19] 10Analytics, 10Analytics-Dashiki, 10Analytics-Kanban, 10Patch-For-Review: Add pivot parameter to tabular layout graphs - https://phabricator.wikimedia.org/T126279#4178415 (10CCicalese_WMF) @Milimetric That's great! Thank you! I will update my graphs as soon as I get a chance. [14:10:11] 10Analytics-Kanban, 10Analytics-Wikimetrics: Wikimetrics database list fetching is broken - https://phabricator.wikimedia.org/T193742#4178422 (10Milimetric) [14:10:23] 10Analytics-Kanban, 10Analytics-Wikimetrics: Wikimetrics database list fetching is broken - https://phabricator.wikimedia.org/T193742#4178434 (10Milimetric) p:05Triage>03Unbreak! [14:25:56] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Wikimedia-Logstash, and 2 others: EventBus HTTP Proxy service does not report errors to logstash - https://phabricator.wikimedia.org/T193230#4178481 (10Ottomata) I tried the other lib real quick too, and didn't have it work. I just need to get back into it. [15:07:44] hi all! :) Just trying to check this out: superset.wikimedia.org [15:08:27] On both Chromium and Firefox, I get "superset.wikimedia.org redirected you too many times." [15:09:17] SoS notes sez, "wikitech account + ping to analytics needed" ... no authentication issues I can see, tho [15:14:05] Same when I try https://superset.wikimedia.org/users/list/, as per https://wikitech.wikimedia.org/wiki/Analytics/Systems/Superset [15:14:15] 10Analytics-Kanban, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), 10Patch-For-Review: Record and aggregate page previews - https://phabricator.wikimedia.org/T186728#4178630 (10mforns) @Tbayer > Instead of creatively reinterpreting this task, can we please just read it as written? It was... [15:15:16] ottomata: ^ (bugging u since u authored the wikitech page) [15:15:21] thx in advance [15:15:41] AndyRussG: i think that happens whenn you don't have an account [15:15:42] looking! [15:15:51] thx! :) [15:17:07] AndyRussG: try now [15:19:07] ottomata: ok now I see a welcome page with a ton of "invalid login" notices and a few "access denied"s interspersed among them [15:20:10] Hmm ok on reload now they're gone [15:20:37] just a list of dashboards now [15:25:06] great! [15:27:12] ottomata: yeah now found the banner activity druid datasource [15:27:22] so, just fiddling with the ui! looks good :) [15:27:25] thx much! [15:27:43] aaarg silly pivot folks, why would they do such a terrible deed? [15:28:18] they had no choice! [15:28:56] AndyRussG: https://github.com/allegro/turnilo was recently noticed! so maybe it is not dead after all.......no promised! :p [15:30:16] hmmm well pretty unfortunate that someone forced their hand then [15:30:20] Hmmm! [15:31:08] yea so far superset seems more powerful tho slower ui than Pivot [15:32:26] yea looks promising :) [15:44:29] 10Analytics, 10Operations: dbstore1002 disk 5 not healthy - https://phabricator.wikimedia.org/T193738#4178362 (10RobH) This has fallen out of warranty as of 2017-02-25, any failed disks (like the one for this task) will need to be replaced from shelf spares. [15:45:57] (03CR) 10Mforns: Add snapshot to datasource-name (mw hist reduced) (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/429770 (https://phabricator.wikimedia.org/T193388) (owner: 10Joal) [15:46:26] (03CR) 10Mforns: "Aaaaah! it's in the other change :]" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/429770 (https://phabricator.wikimedia.org/T193388) (owner: 10Joal) [15:51:09] (03CR) 10Mforns: [C: 031] "LGTM! Left some questions" (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/405053 (owner: 10Joal) [15:51:20] (03CR) 10Mforns: [C: 031] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/429770 (https://phabricator.wikimedia.org/T193388) (owner: 10Joal) [15:53:36] Nice that superset can do more involved filtering [15:54:01] \o/ [15:54:56] ottomata: o/ - if you have 10 mins later on I'd need to ask you a couple of questions about the druid package [15:55:08] maybe after grooming? [15:56:24] elukey: sure! [15:56:31] wanna do now real quick? [15:56:40] i migth need some kafka ssl brainbounce stuff in deployment prep, stuff is weirddddd [15:57:24] ottomata: sure, bc? [15:57:27] ya' [16:00:54] fdans: yohoo [16:01:33] 10Analytics, 10Operations: dbstore1002 disk 5 not healthy - https://phabricator.wikimedia.org/T193738#4178835 (10RobH) p:05Triage>03High [16:05:42] 10Analytics, 10Operations: dbstore1002 disk 5 not healthy - https://phabricator.wikimedia.org/T193738#4178844 (10Marostegui) Not really sure if this is high priority. The disk hasn't failed yet even [16:15:52] 10Analytics: Add legacy per-article pagecounts data (prior to 2015) - https://phabricator.wikimedia.org/T193759#4178888 (10diego) [16:22:42] 10Analytics: Add legacy per-article pagecounts data (prior to 2015) - https://phabricator.wikimedia.org/T193759#4178871 (10MusikAnimal) Thanks for creating this! I think it might be covered by T173720 [16:48:55] 10Analytics, 10Cassandra, 10Maps-Sprint, 10Operations, and 4 others: Upgrade prometheus-jmx-exporter on all services using it - https://phabricator.wikimedia.org/T192948#4179003 (10RobH) p:05Triage>03Normal I'm not quite sure if this is a normal or a high priority task. Seems normal, since we aren't r... [16:51:33] 10Analytics-Kanban, 10EventBus, 10Services (done): Enable snappy compression for eventbus Kafka producer - https://phabricator.wikimedia.org/T193080#4179029 (10Pchelolo) 05Open>03Resolved We've enabled it for change-prop instances as well, so I consider this task resolved. [17:06:22] ping mforns joal milimetric elukey , let's start grooming at :15 after the hour if you guys do not mind as ottomata and I need to take a few minutes to talk [17:07:16] k nuria_ [17:09:29] 10Analytics: Add legacy per-article pagecounts data (prior to 2015) - https://phabricator.wikimedia.org/T193759#4179085 (10Milimetric) [17:09:32] 10Analytics: Add pagecounts by article and top pagecounts to AQS - https://phabricator.wikimedia.org/T173720#4179084 (10Milimetric) [17:10:37] 10Analytics: Add legacy per-article pagecounts data (prior to 2015) - https://phabricator.wikimedia.org/T193759#4178871 (10Milimetric) I'm adding this as a subtask, so we don't lose track of the nice work done to collect the data, but unfortunately the blocker here is space. We appreciate the computation and cl... [17:11:18] 10Analytics: Add legacy per-article pagecounts data (prior to 2015) - https://phabricator.wikimedia.org/T193759#4179089 (10Milimetric) p:05Triage>03Low [17:12:24] 10Analytics, 10Analytics-Wikistats: Missing stats for Atikamekw Wikipedia on stats.wikimedia.org - https://phabricator.wikimedia.org/T193625#4174383 (10Milimetric) Thanks for the report, we'll fix this with the next run, the project is just not in the list of the ones we import right now. [17:13:05] 10Analytics, 10Analytics-Wikistats: Missing stats for Atikamekw Wikipedia on stats.wikimedia.org - https://phabricator.wikimedia.org/T193625#4179101 (10Milimetric) p:05Triage>03High [17:13:55] 10Analytics, 10Product-Analytics: Assess impact of ua-parser update on core metrics - https://phabricator.wikimedia.org/T193578#4179106 (10Milimetric) p:05Triage>03High [17:15:44] 10Analytics: Expose page previews in Rest pageview API - https://phabricator.wikimedia.org/T192490#4179124 (10Milimetric) p:05Normal>03Triage [17:16:09] 10Analytics: Publish data on seen page previews - https://phabricator.wikimedia.org/T193524#4171981 (10Milimetric) [17:16:10] 10Analytics: Expose page previews in Rest pageview API - https://phabricator.wikimedia.org/T192490#4140572 (10Milimetric) [17:16:18] 10Analytics: Publish data on seen page previews - https://phabricator.wikimedia.org/T193524#4171981 (10Milimetric) p:05Triage>03Normal [17:16:55] 10Analytics: Rename new geowiki to geoeditors - https://phabricator.wikimedia.org/T193429#4179137 (10Milimetric) p:05Triage>03Normal [17:18:07] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Services (watching): Modern Event Platform (with EventLogging of the Future (EoF)) - https://phabricator.wikimedia.org/T185233#3910345 (10Milimetric) [17:22:34] 10Analytics, 10EventBus, 10ORES, 10Reading-Infrastructure-Team-Backlog, and 3 others: Emit revision-score event to EventBus and expose in EventStreams - https://phabricator.wikimedia.org/T167180#4179165 (10Milimetric) p:05Triage>03Normal [17:22:38] 10Analytics, 10Wikimedia-Stream, 10Patch-For-Review: Create /v2/schema/:schema_uri endpoint for eventstreams that proxies schemas from eventbus - https://phabricator.wikimedia.org/T160748#4179169 (10Milimetric) p:05Triage>03Normal [17:22:41] 10Analytics, 10ChangeProp, 10EventBus, 10Services (later): Support per-topic configuration in EventBus service - https://phabricator.wikimedia.org/T157092#4179171 (10Milimetric) p:05Triage>03Normal [17:29:00] mforns: [17:29:01] https://www.irccloud.com/pastebin/CAnfVdck/ [17:29:31] thanks! [17:36:36] 10Analytics, 10Operations: dbstore1002 disk 5 not healthy - https://phabricator.wikimedia.org/T193738#4179270 (10RobH) p:05High>03Normal [17:57:43] Arf - Arrived too late [17:58:04] any decision made a-team on how to handle the swap? [17:58:13] :] [17:58:34] joal: decided to talk tomorrow in standup [17:58:41] ok :) [17:58:52] (I personally think whoever's doing this should just do it however they see best, we've all heard each other's points) [18:02:53] joal: i suggested just to manual puppet for now, and then keep arguing how to make it automatic later :) [18:03:17] this works ottomata :) [18:03:47] 10Analytics, 10Analytics-Wikistats, 10Accessibility, 10Easy, 10Patch-For-Review: Wikistats Beta: Fix accessibility/markup issues of Wikistats 2.0 - https://phabricator.wikimedia.org/T185533#4179413 (10MCornacchio) a:05MCornacchio>03None [18:21:32] elukey: still there? [18:21:43] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Update version of ua-parser in eventlogging - https://phabricator.wikimedia.org/T192529#4179427 (10mforns) [18:22:14] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Update version of ua-parser in eventlogging - https://phabricator.wikimedia.org/T192529#4179429 (10mforns) [18:22:58] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Services (watching): Modern Event Platform (with EventLogging of the Future (EoF)) - https://phabricator.wikimedia.org/T185233#4179430 (10Ottomata) [18:27:12] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Pull wmf ua-parser-core from the upstream repo - https://phabricator.wikimedia.org/T192465#4179436 (10Nuria) [18:27:16] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Update ua-parser package. Both uap-java and uap-core - https://phabricator.wikimedia.org/T192464#4179439 (10Nuria) [18:37:25] 10Analytics, 10EventBus: SSL and inter broker encryption for Kafka main - https://phabricator.wikimedia.org/T193778#4179461 (10Ottomata) [18:40:00] 10Analytics, 10EventBus: SSL and inter broker encryption for Kafka main - https://phabricator.wikimedia.org/T193778#4179478 (10Ottomata) [18:40:47] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Services (watching): Modern Event Platform (with EventLogging of the Future (EoF)) - https://phabricator.wikimedia.org/T185233#4179491 (10MMiller_WMF) [18:50:06] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10Services (watching): Upgrade Kafka on main cluster with security features - https://phabricator.wikimedia.org/T167039#4179515 (10Ottomata) [18:51:50] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10Services (watching): Upgrade Kafka on main cluster with security features - https://phabricator.wikimedia.org/T167039#3315367 (10Ottomata) [18:51:52] 10Analytics, 10EventBus: SSL and inter broker encryption for Kafka main - https://phabricator.wikimedia.org/T193778#4179461 (10Ottomata) [18:53:50] 10Analytics, 10Analytics-Wikimetrics: Wikimetrics docker build/test environment is broken - https://phabricator.wikimedia.org/T193780#4179521 (10Milimetric) [18:56:48] What was Idoing in services chan milimetric? weird... [18:56:56] :) [18:56:57] Gone for tonight team -xee you tomorrow [18:57:00] I just went with it [18:57:02] have a nice night [18:58:51] (03PS1) 10Milimetric: Fix fetching the list of databases [analytics/wikimetrics] - 10https://gerrit.wikimedia.org/r/430643 (https://phabricator.wikimedia.org/T193742) [19:00:02] (03CR) 10Milimetric: [V: 032 C: 032] "I'm going to merge this for now, we made all the reasonable attempts to contact everyone and get their permission." [analytics/wikimetrics] - 10https://gerrit.wikimedia.org/r/398412 (https://phabricator.wikimedia.org/T60753) (owner: 10Milimetric) [19:00:07] (03CR) 10jerkins-bot: [V: 04-1] Fix fetching the list of databases [analytics/wikimetrics] - 10https://gerrit.wikimedia.org/r/430643 (https://phabricator.wikimedia.org/T193742) (owner: 10Milimetric) [19:00:33] (03CR) 10Milimetric: [V: 032 C: 032] "merging this just to test, follow-ups may be required since the local testing environment is broken." [analytics/wikimetrics] - 10https://gerrit.wikimedia.org/r/430643 (https://phabricator.wikimedia.org/T193742) (owner: 10Milimetric) [19:03:13] 10Analytics, 10Analytics-Wikimetrics: Wikimetrics deploy is broken - https://phabricator.wikimedia.org/T193783#4179585 (10Milimetric) [19:03:23] jesus everything about Wikimetrics infrastructure seems broken [19:04:08] spring cleaning [19:05:01] I guess... I can test in production? [19:18:44] (03CR) 10Nuria: "Agreed, thanks for merging." [analytics/wikimetrics] - 10https://gerrit.wikimedia.org/r/398412 (https://phabricator.wikimedia.org/T60753) (owner: 10Milimetric) [19:19:55] !log wikimetrics is partly broken until I can figure out what’s going on [19:19:57] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:48:46] ottomata: I'm kind of lost with systemd and wikimetrics, I need a little help [19:50:35] milimetric: sure [19:50:36] what's up? [19:50:56] I can't figure out how to get to the logs [19:51:02] on wikimetrics-01.eqiad.wmflabs [19:51:03] I do journalctl -f -u uwsgi-wikimetrics-web [19:51:21] I get nothing [19:51:34] if I do systemctl uwsgi-wikimetrics-web, that seems fine [19:51:47] like status running everything fine [19:51:59] I also tried with .service [19:55:01] ottomata: am I doing something wrong or... is something else wrong... ? [19:57:41] looking [19:58:44] milimetric: sudo? [19:58:45] sudo journalctl -f -u uwsgi-wikimetrics-web [19:58:45] ? [19:58:50] omg [19:59:07] :) [19:59:35] yep [20:00:09] it could've said access denied like a decent human being that it's not [20:03:56] 10Analytics, 10Product-Analytics, 10Reading-analysis: Assess impact of ua-parser update on core metrics - https://phabricator.wikimedia.org/T193578#4179819 (10MBinder_WMF) [20:14:03] (03PS1) 10Milimetric: Add new config [analytics/wikimetrics-deploy] - 10https://gerrit.wikimedia.org/r/430670 [20:18:01] (03PS1) 10Milimetric: Fix cohort bug introduced last commit [analytics/wikimetrics] - 10https://gerrit.wikimedia.org/r/430675 [20:18:12] (03CR) 10Milimetric: [V: 032 C: 032] Add new config [analytics/wikimetrics-deploy] - 10https://gerrit.wikimedia.org/r/430670 (owner: 10Milimetric) [20:18:19] (03CR) 10Milimetric: [V: 032 C: 032] Fix cohort bug introduced last commit [analytics/wikimetrics] - 10https://gerrit.wikimedia.org/r/430675 (owner: 10Milimetric) [20:20:25] 10Analytics, 10Analytics-Wikimetrics: Wikimetrics deploy is broken - https://phabricator.wikimedia.org/T193783#4179585 (10Milimetric) [20:29:29] !log fixed wikimetrics issues, working fine again [20:29:30] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:43:16] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4180059 (10Psychoslave) [20:45:07] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4178076 (10Psychoslave) [23:00:14] 10Analytics, 10Operations, 10decommission, 10Patch-For-Review: Decommission stat1002.eqiad.wmnet - https://phabricator.wikimedia.org/T173097#4180562 (10RobH) [23:06:12] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4180593 (10Denny) You state that "there are contributors of Wikidata that do make massive imports of external data banks, regardless of the corresponding terms of use." Can you point t...