[07:55:30] !log restart pageview-hourly oozie coordinator to pick up new hive2 action settings [07:55:32] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:02:37] (03CR) 10Elukey: [C: 03+1] Making error message on refine monitor more precise (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/524536 (https://phabricator.wikimedia.org/T228522) (owner: 10Nuria) [08:25:50] 10Quarry: Ask python scripts to use custom user agents - https://phabricator.wikimedia.org/T197258 (10Xqt) Hasn't this been done by rPWBC9a20435 ? [08:25:58] 10Quarry: Ask python scripts to use custom user agents - https://phabricator.wikimedia.org/T197258 (10Xqt) p:05Triage→03Low [09:03:12] 10Analytics, 10LDAP-Access-Requests, 10Operations, 10wikimediafoundation.org: Access to WikimediaFoundation.org analytics for Deb - https://phabricator.wikimedia.org/T227496 (10MoritzMuehlenhoff) 05Resolved→03Open @herron: If you add an account to a PII-relevant LDAP group which does not have shell acc... [09:04:22] 10Analytics, 10Patch-For-Review, 10User-Elukey: Set up a generic workflow to create Kerberos accounts - https://phabricator.wikimedia.org/T226104 (10elukey) [09:04:33] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Set up a generic workflow to create Kerberos accounts - https://phabricator.wikimedia.org/T226104 (10elukey) [09:46:25] 10Analytics, 10Analytics-Kanban, 10Wikimedia-Portals, 10cloud-services-team: https://dumps.wikimedia.org/other/pageviews/ lacks hourly pageviews since 20190722-17:00 - https://phabricator.wikimedia.org/T228731 (10hashar) @JHedden it seems the DNS change has been harmless and the issue came from some of the... [09:48:13] 10Analytics, 10Analytics-Kanban, 10Wikimedia-Portals, 10cloud-services-team: https://dumps.wikimedia.org/other/pageviews/ lacks hourly pageviews since 20190722-17:00 - https://phabricator.wikimedia.org/T228731 (10elukey) 05Open→03Resolved [09:52:12] 10Analytics, 10User-Elukey: Move refinery to hive 2 actions - https://phabricator.wikimedia.org/T227257 (10elukey) @EBernhardson hi! Can we restart the coordinators listed in https://gerrit.wikimedia.org/r/523212 to pick up the new changes? [10:04:46] (03PS1) 10Elukey: aqs: move the oozie hourly coordinator to hive2 actions [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525247 (https://phabricator.wikimedia.org/T227257) [10:06:54] 10Analytics, 10Analytics-EventLogging, 10Operations: Decommission m4 proxies (dbproxy1004 and dbproxy1008) - https://phabricator.wikimedia.org/T228768 (10Marostegui) p:05Triage→03Normal [10:08:43] (03PS1) 10Elukey: banner_activity: move oozie daily coordinator to hive2 actions [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525248 (https://phabricator.wikimedia.org/T227257) [10:10:55] 10Analytics, 10Analytics-EventLogging, 10Operations: Decommission m4 proxies (dbproxy1004 and dbproxy1008) - https://phabricator.wikimedia.org/T228768 (10elukey) +2 from Analytics [10:12:50] 10Analytics, 10Analytics-EventLogging, 10DBA, 10Operations: Decommission m4 proxies (dbproxy1004 and dbproxy1008) - https://phabricator.wikimedia.org/T228768 (10Marostegui) a:03Marostegui Great - thanks. I will get them decommissioned [10:15:48] 10Analytics, 10Analytics-EventLogging, 10DBA, 10Operations: Decommission dbproxy1004 and dbproxy1009 - https://phabricator.wikimedia.org/T228768 (10Marostegui) [10:16:02] 10Analytics, 10Analytics-EventLogging, 10DBA, 10Operations, 10decommission: Decommission dbproxy1004 and dbproxy1009 - https://phabricator.wikimedia.org/T228768 (10Marostegui) [10:29:01] * elukey lunch! [12:21:26] 10Analytics, 10Analytics-EventLogging, 10DBA, 10Operations, 10decommission: Decommission dbproxy1004 and dbproxy1009 - https://phabricator.wikimedia.org/T228768 (10Marostegui) [12:21:54] 10Analytics, 10Analytics-EventLogging, 10DBA, 10Operations, 10decommission: Decommission dbproxy1004 and dbproxy1009 - https://phabricator.wikimedia.org/T228768 (10Marostegui) I have stopped haproxy on both hosts, and will leave it like that for 24h, just to be fully sure nothing uses it. [12:57:34] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Set up a generic workflow to create Kerberos accounts - https://phabricator.wikimedia.org/T226104 (10elukey) I am debugging an issue that Dan faced, and Andrew as well some time ago. Any command on an-tool1006 (like hdfs dfs -ls) leads to: ` Caused by: GSSExce... [13:08:09] (03PS7) 10Fdans: Add UDF to get wiki project from referer string [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/523903 (https://phabricator.wikimedia.org/T228151) [13:11:08] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Set up a generic workflow to create Kerberos accounts - https://phabricator.wikimedia.org/T226104 (10Ottomata) You did something to fix this for me before, what was it? [13:17:09] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Set up a generic workflow to create Kerberos accounts - https://phabricator.wikimedia.org/T226104 (10elukey) >>! In T226104#5361106, @Ottomata wrote: > You did something to fix this for me before, what was it? IIRC I simply re-created the user, and thought tha... [13:19:23] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Set up a generic workflow to create Kerberos accounts - https://phabricator.wikimedia.org/T226104 (10elukey) My principal was working fine on an-tool1006, I have deleted and re-created it, same problem as Dan's. [13:19:49] (03PS1) 10Fdans: Release 2.6.3 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/525279 [13:20:40] (03CR) 10Fdans: [V: 03+2 C: 03+2] Release 2.6.3 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/525279 (owner: 10Fdans) [13:22:46] (03PS1) 10Bearloga: Hash temporary identifiers in app schemas [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525280 (https://phabricator.wikimedia.org/T226852) [13:28:50] ottomata: nuria just checked every single reportupdater job and none stopped working when we made the switch, so I think it's safe to merge the removal tasks [13:29:29] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Set up a generic workflow to create Kerberos accounts - https://phabricator.wikimedia.org/T226104 (10MoritzMuehlenhoff) Is this limited to an-tool1006 or also other hosts? Is this limited to the HDFS command or are other commands also affected? Do basic operati... [13:37:14] (03PS1) 10Ottomata: Clarify RefineMonitor report message [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/525283 (https://phabricator.wikimedia.org/T228522) [13:38:38] fdans: alrighg! [13:39:41] (03CR) 10Elukey: Clarify RefineMonitor report message (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/525283 (https://phabricator.wikimedia.org/T228522) (owner: 10Ottomata) [13:41:18] (03CR) 10Ottomata: Clarify RefineMonitor report message (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/525283 (https://phabricator.wikimedia.org/T228522) (owner: 10Ottomata) [13:44:48] (03CR) 10Elukey: [C: 03+1] Clarify RefineMonitor report message (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/525283 (https://phabricator.wikimedia.org/T228522) (owner: 10Ottomata) [13:53:12] 10Analytics, 10Analytics-Kanban, 10Discovery, 10Operations, 10Research-Backlog: Make oozie swift upload emit event to Kafka about swift object upload complete - https://phabricator.wikimedia.org/T227896 (10Ottomata) But hm, I get your point. It might be nice if the upload script automated some versionin... [13:53:41] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Set up a generic workflow to create Kerberos accounts - https://phabricator.wikimedia.org/T226104 (10elukey) >>! In T226104#5361181, @MoritzMuehlenhoff wrote: > Is this limited to an-tool1006 or also other hosts? > Is this limited to the HDFS command or are oth... [14:09:23] mforns: regarding https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/EventLogging/+/524575/3/modules/ext.eventLogging/Queue.js [14:09:38] there's a while queue.length loop above [14:09:59] I didn't try it but thought that should work [14:10:01] * milimetric tries [14:10:21] fdans: am I a reviewer for removal change? [14:13:25] fdans: am available to take action when you are! (just not remembering what next action is :p ) [14:25:54] milimetric, sorry for that, missed the while [14:28:29] fdans: hola! [14:28:37] fdans: sounds good, let's message all users [14:28:40] mforns: no, thanks for the review. btw, I don't have anything new since yesterday but I'm around if you want to brainbounce together [14:28:53] nuria: cool! [14:29:07] milimetric, me neither, ok [14:29:34] mforns: oh ok, then let's think on it some more, I'm gonna run some more queries [14:29:39] but I'm around working on it if you want to chat [14:36:08] (03CR) 10Nuria: [C: 03+2] Add UDF to get wiki project from referer string [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/523903 (https://phabricator.wikimedia.org/T228151) (owner: 10Fdans) [14:37:23] (03CR) 10Mforns: Hash temporary identifiers in app schemas (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525280 (https://phabricator.wikimedia.org/T226852) (owner: 10Bearloga) [14:39:45] (03CR) 10Nuria: [C: 03+2] "Looks good, merged and noted on https://etherpad.wikimedia.org/p/analytics-weekly-train that aqs hourly job needs to re-start" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525247 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [14:41:21] (03CR) 10Nuria: [C: 03+2] Making error message on refine monitor more precise (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/524536 (https://phabricator.wikimedia.org/T228522) (owner: 10Nuria) [14:42:30] (03CR) 10Nuria: [C: 03+2] Clarify RefineMonitor report message [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/525283 (https://phabricator.wikimedia.org/T228522) (owner: 10Ottomata) [14:43:30] ottomata: much better message and after seeing it Iam now 99% sure that my theory of what happened the other day is correct regarding sanitize [14:43:34] ottomata: re https://gerrit.wikimedia.org/r/#/c/analytics/refinery/source/+/525283/ [14:43:37] cc elukey [14:44:23] (03CR) 10Nuria: [C: 03+2] banner_activity: move oozie daily coordinator to hive2 actions [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525248 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [14:45:03] (03CR) 10Nuria: [C: 03+2] "Added note to https://etherpad.wikimedia.org/p/analytics-weekly-train about this job needed to be re-started" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525248 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [14:45:10] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops: Allow analytics VLAN to reach eventgate-analytics.discovery.wmnet:31192 - https://phabricator.wikimedia.org/T228882 (10Ottomata) [14:46:47] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Set up a generic workflow to create Kerberos accounts - https://phabricator.wikimedia.org/T226104 (10elukey) Found something interesting. If I create the user with `+needchange`, I get the issue; without the flag, all HDFS commands work fine. [14:47:42] (03Merged) 10jenkins-bot: Clarify RefineMonitor report message [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/525283 (https://phabricator.wikimedia.org/T228522) (owner: 10Ottomata) [14:55:55] (03CR) 10Nuria: [C: 04-1] Hash temporary identifiers in app schemas (033 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525280 (https://phabricator.wikimedia.org/T226852) (owner: 10Bearloga) [15:01:04] 10Analytics, 10Analytics-Kanban, 10Discovery, 10Operations, 10Research-Backlog: Make oozie swift upload emit event to Kafka about swift object upload complete - https://phabricator.wikimedia.org/T227896 (10Nuria) @Ottomata I think for users sake it is easier to do it the other way around maybe? Provide v... [15:10:30] trying to add a datasource in superset, but when i `sync columns from source` i get 401 unauthorized :S any ideas? [15:10:30] 10Analytics: mediawiki-history-wikitext-coord job fails every month - https://phabricator.wikimedia.org/T228883 (10Nuria) [15:11:18] 10Analytics, 10Pageviews-API, 10Tool-Pageviews: 429 Too Many Requests hit despite throttling to 100 req/sec - https://phabricator.wikimedia.org/T219857 (10Milimetric) This can be tricky to diagnose because we don't really know what if any upstream changes are made to Hyperswitch. Do you have a more accurate... [15:11:52] ebernhar1son: what kind of datasource? :) [15:12:18] we suggest to use only Druid and Mysql if possible, Hive is not enabled [15:12:25] elukey: its a new druid datasource, search_satisfaction_hourly_wip. [15:12:46] maybe it worked this time, i deleted it and clicked the 'scan new datasources' [15:13:08] ah good :) [15:13:16] I can check logs if it is still an issue [15:14:15] elukey: it looks to have pulled it in. One other problem though, when i click on the https://superset.wikimedia.org/dashboard/new/ link i'm redirected to login, which autoredirects me back to the list of dashboards. [15:14:50] oddly /chart/add works fine [15:14:52] ebernhar1son: there is some DARK MAGIC here in which this error only affects developers in OTHER Teams [15:15:11] ebernhar1son: can you try FF and see if error happpens too? [15:15:18] sure, sec [15:15:40] https://phabricator.wikimedia.org/T224159 [15:15:43] ebernhar1son: let me look at your user in superset [15:16:02] same redirect loop in FF 68 [15:16:53] that ticket looks very similar, i've gotten not every page but seemingly random re-auth [15:17:00] (and some 401 unahtorized api responses) [15:18:02] ebernhar1son: i just changed 1 thing, can you so kindly log out/delete cookies and log back in? [15:18:25] sure, sec [15:19:05] nuria: using an incog window, looks to work! thanks [15:19:19] elukey: i think this related to user roles [15:19:37] ebernhar1son: and try using now your regular browser window [15:19:54] nuria: what did you change? [15:21:08] elukey: i made erik an admin, but let's see if it works in ebernhar1son 's regular browser [15:21:36] ah interesting [15:24:13] milimetric, wanna chat on public data lake? [15:24:23] I'm in a meeting now mforns [15:24:30] ok, let me know [15:24:48] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Core Platform Team (Modern Event Platform (TEC2)), and 3 others: Modern Event Platform (TEC2) - https://phabricator.wikimedia.org/T185233 (10Ottomata) [15:26:02] a-team: Rack A1 is about to go under maintenance, it holds kafka-jumbo1001 and analytics1058. In theory we shouldn't see impact, in practice we might and if so you'll know why :) [15:26:15] ack [15:29:38] ebernhar1son: ping us if you find issues on your regular browser window loginin, i made you an , ahem, admin cause i think there is a bug on auth with "alpha" users (untested theory) so be watchful cause you have all-permits-that-there-are in superset [15:36:48] ottomata: ops sync? [15:39:07] yargh elukey in other meeting [15:39:12] want to do after standup? [15:39:17] sure! [15:39:19] or this meeting might be done in 5 or 10 mins [15:39:22] we can do quick before standup [15:46:06] (brb) [15:48:24] nuria: what was that doc you had against EL being used for client errors? [15:48:31] you put it up on a wiki somewhere [16:02:05] a-team: will be 2 mins late for standup [16:02:26] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10MediaWiki-JobQueue, and 3 others: Migrate JobQueue to eventgate - https://phabricator.wikimedia.org/T228705 (10Ottomata) a:03Pchelolo [16:08:09] 10Analytics, 10Analytics-Kanban: Page creation data stream died June 6 - https://phabricator.wikimedia.org/T228188 (10Ottomata) @kaldari we'd like to stop producing this data to MySQL. We are waiting for your feedback. [16:09:21] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops: Allow analytics VLAN to reach eventgate-analytics.discovery.wmnet:31192 - https://phabricator.wikimedia.org/T228882 (10Ottomata) a:05Ottomata→03None [16:18:34] 10Analytics, 10Better Use Of Data, 10Reading-Infrastructure-Team-Backlog, 10Epic: Client side error logging production launch - https://phabricator.wikimedia.org/T226986 (10Milimetric) Wanted to mention this in today's meeting but couldn't find it in time: https://wikitech.wikimedia.org/wiki/Analytics/Syst... [16:34:10] 10Analytics, 10Better Use Of Data, 10Reading-Infrastructure-Team-Backlog, 10Epic: Client side error logging production launch - https://phabricator.wikimedia.org/T226986 (10Nuria) Also, per https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging/NotErrorLogging let's please have in mind that ana... [16:34:50] (03CR) 10Ottomata: [V: 03+2 C: 03+2] Delete limn-flow-data queries in favor of reportupdater-queries [analytics/limn-flow-data] - 10https://gerrit.wikimedia.org/r/523731 (https://phabricator.wikimedia.org/T222739) (owner: 10Fdans) [16:34:58] (03CR) 10Ottomata: [V: 03+2 C: 03+2] Delete limn-edit-data queries in favor of reportupdater-queries [analytics/limn-edit-data] - 10https://gerrit.wikimedia.org/r/523732 (https://phabricator.wikimedia.org/T222739) (owner: 10Fdans) [16:35:05] (03CR) 10Ottomata: [V: 03+2 C: 03+2] Delete limn-language-data queries in favor of reportupdater-queries [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/523733 (https://phabricator.wikimedia.org/T222739) (owner: 10Fdans) [16:35:11] (03CR) 10Ottomata: [V: 03+2 C: 03+2] Delete limn-ee-data queries in favor of reportupdater-queries [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/523734 (https://phabricator.wikimedia.org/T222739) (owner: 10Fdans) [16:35:13] (03Merged) 10jenkins-bot: Delete limn-language-data queries in favor of reportupdater-queries [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/523733 (https://phabricator.wikimedia.org/T222739) (owner: 10Fdans) [16:39:40] 10Analytics, 10Better Use Of Data, 10Reading-Infrastructure-Team-Backlog, 10Epic: Client side error logging production launch - https://phabricator.wikimedia.org/T226986 (10Tgr) Per today's meeting, next steps on the client side (alongside with defining the EventGate schema) is to write a minimal client th... [16:40:03] fdans: semantic has placeholders... not sure I love them: v [16:40:04] https://semantic-ui.com/introduction/new.html [16:40:33] !log removed all non reportupdater-queries job repositories from /srv/reportupdater/jobs/ - T222739 [16:40:37] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:40:37] T222739: Move reportupdater queries from limn-* repositories to reportupdater-queries - https://phabricator.wikimedia.org/T222739 [16:40:57] milimetric: ohhh I don't completely hate them [16:41:06] but it's semantic so I'll hate implementing them I'm sure [16:57:55] 10Analytics, 10Better Use Of Data, 10Reading-Infrastructure-Team-Backlog, 10Epic: Client side error logging production launch - https://phabricator.wikimedia.org/T226986 (10Ottomata) From what I can tell, most of the objections on EventLogging/NotErrorLogging are about EventLogging specific stuff. Event P... [17:01:01] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Add new mediatypes to media classification refinery code - https://phabricator.wikimedia.org/T225911 (10Nuria) [17:10:40] 10Analytics, 10LDAP-Access-Requests, 10Operations, 10wikimediafoundation.org, 10Patch-For-Review: Access to WikimediaFoundation.org analytics for Deb - https://phabricator.wikimedia.org/T227496 (10RStallman-legalteam) I don't actually see the paper work for WMF full time req # employees, so I think havin... [17:13:54] * elukey off! [17:20:05] 10Analytics, 10LDAP-Access-Requests, 10Operations, 10wikimediafoundation.org: Access to WikimediaFoundation.org analytics for Deb - https://phabricator.wikimedia.org/T227496 (10Heather) It doesn't seem like you need it, but this is approved. Let me know if you need something else. [17:23:52] 10Analytics, 10MobileFrontend, 10Readers-Web-Backlog: Having trouble setting up MobileFrontend for development - https://phabricator.wikimedia.org/T226071 (10ovasileva) [17:30:07] 10Analytics, 10LDAP-Access-Requests, 10Operations, 10wikimediafoundation.org: Access to WikimediaFoundation.org analytics for Deb - https://phabricator.wikimedia.org/T227496 (10herron) 05Open→03Resolved Great! Thanks all [17:34:45] 10Analytics, 10Analytics-Kanban, 10JavaScript: Fix the analytics/wikistats2 repo to work on node10 - https://phabricator.wikimedia.org/T228452 (10Jdforrester-WMF) 05Resolved→03Open [17:48:56] 10Analytics, 10Analytics-Kanban, 10Discovery, 10Operations, 10Research-Backlog: Make oozie swift upload emit event to Kafka about swift object upload complete - https://phabricator.wikimedia.org/T227896 (10EBernhardson) I hadn't previously thought about re-publishing a new version of the same dataset. It... [17:49:58] 10Analytics, 10Analytics-Kanban, 10Discovery, 10Operations, 10Research-Backlog: Make oozie swift upload emit event to Kafka about swift object upload complete - https://phabricator.wikimedia.org/T227896 (10Ottomata) Ya, if you needed to re-run a job due to data backfill, you might want to be able to do s... [18:58:06] (03PS1) 10Nuria: Changelog for 0.0.95 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/525343 [19:17:14] (03PS1) 10Milimetric: Update Semantic to work with Node 10 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/525344 (https://phabricator.wikimedia.org/T228452) [19:17:39] (03CR) 10Milimetric: "check experimental" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/525344 (https://phabricator.wikimedia.org/T228452) (owner: 10Milimetric) [19:18:53] 10Analytics, 10Better Use Of Data, 10Reading-Infrastructure-Team-Backlog, 10Epic: Client side error logging production launch - https://phabricator.wikimedia.org/T226986 (10Milimetric) my fault - I confused this with our mediawiki-storage repo, I should've read the title more carefully. Will work on fixing. [19:30:15] (03CR) 10Nuria: [C: 03+2] Changelog for 0.0.95 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/525343 (owner: 10Nuria) [19:30:30] (03CR) 10Nuria: [C: 03+2] "Self merging per deploy protocol." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/525343 (owner: 10Nuria) [19:32:03] 10Analytics, 10Better Use Of Data, 10Reading-Infrastructure-Team-Backlog, 10Epic: Client side error logging production launch - https://phabricator.wikimedia.org/T226986 (10Nuria) I would like to suggest a deployment strategy for this code that I think would make things simple for an MVP (feel free to disr... [19:36:00] (03Merged) 10jenkins-bot: Changelog for 0.0.95 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/525343 (owner: 10Nuria) [19:42:08] fdans: i think the last deploy to wikistats broke editors metrics, please see: https://stats.wikimedia.org/v2/#/gl.wikipedia.org/contributing/editors/normal|line|all|~total|monthly [19:43:02] 10Analytics: wikistats editor graphs broken - https://phabricator.wikimedia.org/T228931 (10Nuria) [19:43:21] 10Analytics: wikistats editor graphs broken - https://phabricator.wikimedia.org/T228931 (10Nuria) {F29862252} [19:43:59] 10Analytics, 10Analytics-Kanban: Deal with truncated values in uniques - https://phabricator.wikimedia.org/T220098 (10Nuria) Reverting this change as it broke editors charts [19:45:13] fdans: I am going to revert change , please see: https://phabricator.wikimedia.org/T228931 [19:49:30] milimetric: yt? [19:50:43] nuria: yea but grooming tech com and meeting starts in 10 [19:51:25] milimetric: ok, i am going to try to fix wikistats which is broken after latest deploy , will send you CR [19:51:37] milimetric: https://phabricator.wikimedia.org/T228931 [19:53:49] heya team :] anyone remembers why/when we choose to use kryoSerialization in spark? [19:54:59] (03CR) 10Nuria: Add zeroes to truncated values and UI about truncation (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/519382 (https://phabricator.wikimedia.org/T220098) (owner: 10Fdans) [19:57:32] mforns: did we choose explicitly? [19:58:10] ottomata, in the clickstream job we do I think [19:58:50] and as it is the only job I found that uses gzip format with repartitioning, I wonder if we need kryoSerialization for that [19:59:13] cause I'm trying to use gzip+repartitioning when writing [20:06:13] (03PS1) 10Nuria: Correcting check for truncated values [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/525360 (https://phabricator.wikimedia.org/T228931) [20:06:36] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: wikistats editor graphs broken - https://phabricator.wikimedia.org/T228931 (10Nuria) [20:06:51] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: wikistats editor graphs broken - https://phabricator.wikimedia.org/T228931 (10Nuria) ping @fdans will try to fix and deploy this today [20:07:41] milimetric:, mforns if any of you can CR this https://gerrit.wikimedia.org/r/#/c/analytics/wikistats2/+/525360/ I can deploy and correct wikistats which is bropen in prod [20:07:51] *broken [20:09:45] (03CR) 10Bearloga: Hash temporary identifiers in app schemas (034 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525280 (https://phabricator.wikimedia.org/T226852) (owner: 10Bearloga) [20:10:22] (03PS2) 10Bearloga: Hash temporary identifiers in app schemas [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525280 (https://phabricator.wikimedia.org/T226852) [20:54:42] ping mforns or milimetric [20:55:10] nuria, yea? [20:55:51] 10Analytics: Add agent type split to wikistats pageviews - https://phabricator.wikimedia.org/T228937 (10Nuria) [20:56:45] mforns: can you take a look a https://gerrit.wikimedia.org/r/#/c/analytics/wikistats2/+/525360/, wikistats is broken in prod for metric whose values are on the thousand range and i think this would fix issue [20:57:39] nuria, how can I repoduce the error? [20:58:39] mforns: see https://phabricator.wikimedia.org/T228931 [21:00:10] ok, nuria, I'm out of my meeting now [21:00:17] I'll take a look too [21:00:20] milimetric: k! [21:00:42] nuria, I can not reproduce in prod though [21:00:49] https://stats.wikimedia.org/v2/#/gl.wikipedia.org/contributing/editors/normal|line|all|~total|monthly [21:00:57] oh, now I can [21:01:19] hm, this looks pretty bad: https://stats.wikimedia.org/v2/#/en.wikipedia.org/reading/total-page-views/normal|bar|3-month|~total|daily [21:03:47] 10Analytics, 10Better Use Of Data, 10Reading-Infrastructure-Team-Backlog, 10Epic: Client side error logging production launch - https://phabricator.wikimedia.org/T226986 (10Nuria) Hawain wikipedia has about 5000K pageviews daily, about 3000 from users with a good representation of IE11 (20%) Even if we ha... [21:03:56] which issue are we talking about though? I can see 2 in: https://stats.wikimedia.org/v2/#/gl.wikipedia.org/contributing/editors/normal|line|all|~total|monthly [21:04:13] 1) the annotation is floating and looks weird [21:04:36] mforns, milimetric they are both the same one I think, due to this check: https://gerrit.wikimedia.org/r/#/c/analytics/wikistats2/+/525360/ [21:04:43] 2) you can still see the hidden values if you hover over the part that is over the threshold [21:05:11] yeah, this is all kinds of broken, I would just revert to the last version and fix it properly [21:05:21] I'm sorry I messed up this code review [21:05:23] mforns:, milimetric : the 1000 limit i think is causing both issues [21:05:28] I misunderstood the impact on the UI [21:05:55] nuria: yeah, but I tried applying your patch and it doesn't change it, because those metrics have truncated values [21:06:01] mforns, milimetric : because in the metrics like unique devices that do not report data for <1000 data is not sent [21:06:11] mforns, milimetric : thus no issues with mouseovers [21:06:30] oh, no, you're right [21:06:34] mforns, milimetric : also bizarre split does not happen [21:06:47] I was looking at a cached version, sorry [21:07:05] milimetric, mforns : so fix is simple i think, but by all means let me know if you disagree [21:08:30] nuria, so is the objective of that change to make the threshold disappear? [21:08:37] (03CR) 10Milimetric: [C: 03+2] Correcting check for truncated values [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/525360 (https://phabricator.wikimedia.org/T228931) (owner: 10Nuria) [21:08:50] mforns, milimetric : for the metrics other than unique-devices, yes [21:08:56] ok ok [21:09:04] ok, merged [21:09:15] fyi: if you want to see the limit, you can see it here: bg.wikinews.org/reading/unique-devices/normal|line|2-year|~total|monthly [21:09:21] (after nuria's patch) [21:09:55] milimetric, mforns : thank you, need to go somewhere where I can plug my computer, will deploy in a bit [21:10:09] ok! [21:10:49] (03PS2) 10Milimetric: Update Semantic to work with Node 10 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/525344 (https://phabricator.wikimedia.org/T228452) [21:11:02] (03CR) 10Milimetric: "check experimental" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/525344 (https://phabricator.wikimedia.org/T228452) (owner: 10Milimetric) [21:11:11] (03Merged) 10jenkins-bot: Correcting check for truncated values [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/525360 (https://phabricator.wikimedia.org/T228931) (owner: 10Nuria) [21:11:34] (that's an unrelated change, nuria, we don't have to +2 it until after you deploy if you want - up to you I'll add you to the review) [21:12:42] milimetric: k [21:15:28] (03PS3) 10Milimetric: Update Semantic to work with Node 10 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/525344 (https://phabricator.wikimedia.org/T228452) [21:15:46] (03PS4) 10Milimetric: Update Semantic and change to Headless for node 10 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/525344 (https://phabricator.wikimedia.org/T228452) [21:16:12] (03CR) 10Milimetric: "check experimental" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/525344 (https://phabricator.wikimedia.org/T228452) (owner: 10Milimetric) [21:16:21] ottomata: again we cannot deploy to 1007, not enough space [21:16:30] 1007?? [21:16:33] that doesn't usually happen there [21:16:43] 1007 has space [21:17:22] nuria^ [21:17:32] ottomata: wait error might not be space but something else, let me re-try [21:17:47] !log deployment of refinery 0.0.95 aborted [21:17:49] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:25:07] (03PS5) 10Milimetric: Update Semantic and change to Headless for node 10 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/525344 (https://phabricator.wikimedia.org/T228452) [21:25:28] I'm persistent if nothing else :) [21:25:41] (03CR) 10Milimetric: "check experimental" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/525344 (https://phabricator.wikimedia.org/T228452) (owner: 10Milimetric) [21:49:56] (03PS1) 10Nuria: Release 2.6.4 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/525430 [21:51:26] (03CR) 10Nuria: [C: 03+2] "Self-merging per deploy protocol" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/525430 (owner: 10Nuria) [21:54:23] (03Merged) 10jenkins-bot: Release 2.6.4 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/525430 (owner: 10Nuria) [22:00:51] (03PS1) 10Ottomata: [WIP] swift-upload.py to handle upload and event emitting [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525435 (https://phabricator.wikimedia.org/T227896) [22:01:05] nuria: FYI, I'm rewriting all the swift upload code in python now [22:01:21] its too difficult to pass info between the shell oozie job and another python event emitting job [22:01:29] there are multiple pieces of info I need [22:01:44] ottomata: then maybe I WILL understand it, +1 [22:01:46] so I bit the bullet and implemented bash env file parsing in python so we can use the swift client [22:02:01] but annoyingly, the pythonb swift client doesn't support uploading directories [22:02:11] all the logic for uploading directories is in their shell CLI code [22:02:16] ottomata: arghhhh [22:02:17] so I have to shell out to actually do the upload [22:02:49] https://github.com/openstack/python-swiftclient/blob/master/swiftclient/shell.py#L1141-L1148 etc. etc. [22:02:59] I filed https://bugs.launchpad.net/python-swiftclient/+bug/1837794 [22:03:03] ottomata: i see [22:03:06] but ¯\_(ツ)_/¯ [22:04:51] ottomata: ya, it is never easy [22:05:11] ottomata: i am onto my 3rd jenkins build [22:05:26] oh why jenkins builds? it wasn't a scap problem? [22:12:19] ottomata: no, i realized the build had failed [22:12:24] ah ok [22:12:27] https://www.irccloud.com/pastebin/xndaUUg6/ [22:12:37] ottomata: i cannot get it to succeed though [22:12:57] ottomata: https://integration.wikimedia.org/ci/job/analytics-refinery-release/194/console [22:13:25] ottomata: i deleted the prior two failed builds [22:13:29] (03PS2) 10Ottomata: [WIP] swift-upload.py to handle upload and event emitting [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525435 (https://phabricator.wikimedia.org/T227896) [22:15:03] something is wrong nuria, i just git pulled and i see 0.0.95in the pom [22:15:09] it it should be -SNAPSHOT [22:15:13] unless i'm in a tag [22:15:52] nuria: a previous build succeded though? [22:15:59] refinery has sthhe 0.0.95 artifacts? [22:16:26] ottomata: no all failed but i trigger the one that does the commit and that suceeded, boy cause that one does not look that artifact is there [22:16:48] ah [22:16:50] ottomata: no only 94, so i guess i can fix poms? [22:16:56] ok let's get the hammer out... [22:16:57] ottomata: https://archiva.wikimedia.org/#artifact/org.wikimedia.analytics.refinery/refinery [22:17:10] ottomata: wait, let me fix poms no? [22:17:11] the snap shot doesn't ussually get uploaded [22:17:38] ottomata: what are you going to do with the hammer then ,ajuas [22:17:44] not sure [22:17:46] but reset things [22:17:53] to how they were before, delete some tags [22:18:03] i don't know 100% it will work, but it usually [22:18:03] ottomata: i think all is needed is to 1) delete tags (if 95) [22:18:04] doess [22:18:06] want to bc? [22:18:10] nuria: somethign is wrong with master [22:18:20] master shouldn't have the version set to something not -SNAPSHOT [22:18:21] 2) undo last master changeset [22:18:29] ottomata: k, bc [22:18:49] 10Analytics, 10Analytics-Kanban: Page creation data stream died June 6 - https://phabricator.wikimedia.org/T228188 (10kaldari) @Ottomata - Thanks for investigating! I don't really care where the data lives, I just need a publicly-accessible way to monitor page creation trends. This dashboard was doing a great... [22:19:34] 10Analytics, 10Analytics-Kanban: Page creation data stream died June 6 - https://phabricator.wikimedia.org/T228188 (10Ottomata) @kaldari, is the dashboard Nuria linked to in https://phabricator.wikimedia.org/T228188#5347025 good? [22:24:59] (03PS1) 10Nuria: Reverting jenkins faulty commit [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/525446 [22:29:56] (03Abandoned) 10Nuria: Reverting jenkins faulty commit [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/525446 (owner: 10Nuria) [22:31:18] (03PS1) 10Nuria: Revert lastest commit by jenkins [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/525447 [22:32:16] (03CR) 10Ottomata: [V: 03+2 C: 03+2] Revert lastest commit by jenkins [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/525447 (owner: 10Nuria) [22:34:35] 10Analytics, 10Analytics-Kanban: Page creation data stream died June 6 - https://phabricator.wikimedia.org/T228188 (10Nuria) https://stats.wikimedia.org/v2/#/et.wikipedia.org/contributing/new-pages/normal|bar|2-year|page_type~content|monthly and https://stats.wikimedia.org/v2/#/et.wikipedia.org/contributing/ne... [22:42:36] ottomata: build failed again [22:42:50] ottomata: this time error makes sense : [22:42:54] https://www.irccloud.com/pastebin/GIXmayfj/ [22:43:13] ottomata: maybe archiva password has chnaged [22:43:33] HM [22:44:09] ottomata: build: https://integration.wikimedia.org/ci/job/analytics-refinery-release/195/console [22:48:01] indeed nuria i can't log into archiva with archiva-ci creds [22:48:04] will change pw [22:48:10] ottomata: k [22:49:40] !Log uploading of refinery-0.0.95 to archiva failed, reseting archiva pw [22:53:31] ok password changed nuria [22:53:34] and updated in jenkins [22:53:40] i wonder if there is an expiry in archiva or something [22:53:40] !log uploading of refinery-0.0.95 to archiva failed, reseting archiva pw [22:53:42] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:53:55] ottomata: ok, so i can trigger another build, right? [22:54:02] yes [22:54:08] fingers crossed [22:54:09] ottomata: i am going to go with 0.0.96 [22:54:25] ottomata: cause commit history looks a bit bad with too many reverts, sounds ok? [22:57:54] sure whatever [22:57:57] or we could FORCE PUSH [22:57:58] heheh [22:57:59] no go ahead [22:58:01] doesn't matter [22:58:04] i've done it that way before too [22:58:16] i wonder if we should bump something up a .100...:p [23:03:52] 10Analytics, 10Analytics-Kanban: Page creation data stream died June 6 - https://phabricator.wikimedia.org/T228188 (10kaldari) That interface is more limited, but I suppose it will do. Thanks. [23:10:01] 10Analytics, 10Analytics-Kanban: Page creation data stream died June 6 - https://phabricator.wikimedia.org/T228188 (10Nuria) @kaldari it is our plan to work a bit more on the wikistats UI this year, the api has the ability of setting up a combination of "splits" (by "editor type" but also "content type") so it... [23:17:19] ottomata: i think we also need credentials for scap for achiva. I am getting this error https://etherpad.wikimedia.org/p/nuria (see bold) [23:27:59] hm no shouldn't nuria, archiva is open for reads [23:28:21] that just looks like the git fat symlink to the artifacts don't exist... [23:28:22] looking [23:29:06] oh [23:29:22] nuria, its because the refinery artifacts were added previosuly from the bad deploy [23:29:32] i think... [23:29:36] ottomata: added to where? [23:29:40] refinery git [23:29:52] commit 58e64c100d68e713ddeec6d6224778a1ab82574b [23:30:02] ottomata: but the sha should be different no? well i gues snot [23:30:08] 0.0.95 doesn't exist [23:30:11] but it was added [23:30:14] we just have to remove them manuually [23:30:16] making commit... [23:30:27] ottomata: k [23:31:10] (03PS1) 10Ottomata: Remove non-existing version 0.0.95 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525465 [23:31:49] (03CR) 10Ottomata: [V: 03+2 C: 03+2] Remove non-existing version 0.0.95 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525465 (owner: 10Ottomata) [23:31:55] nuria: try now [23:32:45] ottomata: grasias , trying [23:47:04] ottomata: and it finished wowow ~~~~~=> the wave