[00:01:40] 10Analytics, 10Product-Analytics: [BUG] User agent parsing error for MobileWikiAppSearch table - https://phabricator.wikimedia.org/T211833 (10chelsyx) [00:21:01] 10Analytics, 10Analytics-Wikimetrics: Sunset Wikimetrics - https://phabricator.wikimedia.org/T211835 (10Nuria) [00:23:13] 10Analytics: Kerberos Stand Alone service running in production - https://phabricator.wikimedia.org/T211836 (10Nuria) [00:54:50] 10Analytics: Kerberos stand alone service running in production - https://phabricator.wikimedia.org/T211836 (10Nuria) [07:10:04] 10Analytics, 10Operations, 10ops-eqiad, 10User-Elukey: Check analytics1037 power supply status - https://phabricator.wikimedia.org/T179192 (10elukey) This server is going to be decommed very soon (OOW), I've acked the alarm a long time ago to avoid it spamming us. Good to close in my opinion, +1 [07:25:26] fdans: o/ [07:25:28] morningggg [07:25:45] hellooooo luca [07:25:45] I am trying to spin up a superset instance in labs [08:05:27] elukey: how's that going? [08:09:42] fdans: I was able to spin up the current version of superset, tried to cherry pick my patch and deploy but it doesn't seem working, checking now [08:10:29] elukey: if you need to brainbounce or rubberduck I'm here! [08:10:34] thanks! [08:10:51] I must have missed something, I can see [08:10:51] 007/bin/bash: /srv/deployment/analytics/superset/venv/bin/gunicorn: No such file or directory [08:10:57] that is really weird :D [08:11:59] but if I manually restart it, it works [08:12:08] mmmmm [08:13:41] ah no sorry it doesn't [08:25:15] ah found the problem! [08:26:28] the page is very slow now, probably due to the derby db [08:26:37] migth be better to add a mariadb instance in there [08:44:40] (03PS2) 10Elukey: [WIP] Release 0.28.1 [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/479249 [08:45:40] (03PS3) 10Elukey: [WIP] Release 0.28.1 [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/479249 [09:03:54] looping the loop - https://twitter.com/PierreJoye/status/1072865391993479169 [09:04:48] https://github.com/apache/incubator-superset/blob/master/UPDATING.md#superset-0280 [09:04:55] "0.28 forces the migration of all dashboards through an automated db migration script." [09:05:01] which one? where? [09:05:06] * elukey hates documentation [09:05:15] oooooffff /o/ [09:27:10] all right added a proper database, it seems working better [09:27:21] fdans: ssh -L 9080:superset.eqiad.wmflabs:9080 superset.eqiad.wmflabs [09:27:33] and then localhost:9080 on your browser [09:27:42] elukey: trying! [09:27:46] this is a bare install of course [09:27:52] if it asks for a pass, admin/admin [09:31:49] * elukey sees snow outside the window [09:34:58] * joal hopes there REALLY is snow outside elukey's window :) [09:35:18] yeah not an imagination [09:35:45] or an eye/brain induced glitches [09:36:06] elukey: This means it's cold where you are :) [09:39:17] joal: how's the weather in your town? [09:39:50] elukey: very rainy since a few days - Heavy rain - But not too cold (actually, getting there since today) [09:43:06] :) [09:43:40] fdans: so from what I can see the bare site works fine, but we might need to create some stuff to test for example if the fix for Yo [09:43:44] YoY works [09:44:07] so I would do something like this: [09:44:21] 1) drop the database and re-deploy the current superset version [09:44:31] 2) create some test users/dashboards [09:44:45] 3) deploy the new version and check the db upgrade procedure [09:44:57] 4) verify that users/dashboards are ok [09:45:00] what do you think? [09:51:06] (in the meantime, I am going to proceed with 1) [09:52:38] elukey: sounds good to me, site works great from what I can see [09:55:00] fdans: do you have a quick way to reproduce the issue that it was discussed in the task? [09:55:09] (I mean creating a dashboard etc..) [09:55:36] elukey: I was making a csv with some values and dates, going to upload it and try to create one of those [09:55:56] nice [09:56:30] fdans: 1) completed [09:59:35] joal: I have deployed to aqs1004 a new nodejs 6.x upgrade (security) that has been running very well up to now from what I can see [09:59:41] ok if I roll it out to all? [09:59:50] elukey: please :) [10:00:12] ack :) [10:00:33] elukey: hmmm, it seems I don't have permission to upload [10:00:41] https://usercontent.irccloud-cdn.com/file/MhF89USe/Screen%20Shot%202018-12-13%20at%2011.00.17%20AM.png [10:01:33] ah! [10:01:49] does it work in prod? [10:01:59] (very ignorant on this side) [10:02:50] elukey: good point, it doesn't work in prod [10:03:35] lovely [10:03:50] same error? [10:08:25] elukey: yes [10:10:07] so the "/app" part at end of /srv/deployment/analytics/superset/venv/lib/python3.5/site-packages/superset [10:10:08] elukey: where can I access this main sql database? [10:10:10] does not exists [10:10:25] fdans: what do you mean? [10:11:27] elukey: ah, thought it was a known thing. The SQL Lab in superset has as default a "main" DB, and I thought I could just insert the data there and use that as source [10:13:09] fdans: I think that there is an option that we don't set to specify an upload dir for csv files [10:17:17] interesting https://github.com/apache/incubator-superset/issues/4287 [10:17:24] add the following to mysql/supertset_config.py [10:17:24] UPLOAD_FOLDER='/tmp/' [10:18:53] fdans: can you retry now in labs? [10:18:57] (to upload a csv) [10:19:08] on it [10:19:18] <3 [10:20:52] elukey: upload worked! trying chart now [10:21:01] ah! [10:21:11] I added the /tmp config basically [10:25:34] elukey: have you looked at this? [10:25:35] http://localhost:9080/static/assets/version_info.json [10:25:54] I'm afraid the viz failed the same way as in prod [10:26:46] fdans: sorry Fran I didn't get the question [10:27:04] elukey: the version shown is 0.26.3 [10:27:33] that's the same as in prod [10:27:50] ah yes this is expected, in 1) I reverted everything to 0.26.3 to allow us to 2) load data and 3) test the upgrade procedure [10:28:39] there is a upgrade db needed this time (they suggest to back up the db first etc..) [10:28:52] so I want to be sure that it works (at least in labs) to reduce the fun in prod [10:30:28] fdans: --^ [10:31:01] ohhh elukey sorry I had misunderstood that [10:31:14] elukey: ok data upload works great :) [10:31:38] super :) [11:05:35] (03PS4) 10Elukey: Release 0.28.1 [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/479249 (https://phabricator.wikimedia.org/T211605) [11:06:38] fdans: how is the testing going [11:06:38] ? [11:13:55] joal: o/ [11:14:16] do you think that I could stop camus to allow some maintenance for hdfs? [11:14:21] I'd need to add two journal nodes [11:14:29] (and then later on remove two) [11:14:38] but I'd prefer to do it in safe mode if possible [11:19:29] elukey: waitwait I'm not sure what I have to test [11:20:57] fdans: in theory the basic functionalities and the bug that triggered all the discussion about moving to 0.28.1 no? [11:21:44] fdans: https://phabricator.wikimedia.org/T210687 [11:22:09] elukey: sorry, I should have been more clear, see 11:25 chat, the visualization fails with the same error as in prod [11:22:40] ahhhhhh [11:22:48] snap I didn't get it, sorry I am stupid [11:22:53] you had already tested it [11:22:54] sigh [11:23:03] all right, upgrading to 0.28.1 [11:23:04] sorryyyyy luca [11:29:07] done! [11:29:39] fdans: mind to check? [11:29:51] elukey: omw [11:31:19] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Upgrade Superset to 0.28.1 - https://phabricator.wikimedia.org/T211605 (10elukey) While testing the `superset db upgrade` command I got: ` (venv) elukey@superset:/srv/deployment/analytics/superset$ superset db upgrade Loaded your LOCAL configuration at [/... [11:36:34] elukey: it seems to work!! [11:36:59] elukey: my data is not great for this viz type but it definitely works [11:37:20] \o/ [11:37:38] are we bold enough to attempt a deployment? [11:41:20] elukey: hell yeaaa [11:41:42] elukey: analysts are still sleeping [11:45:24] fdans: let's do it [11:45:28] so procedure: [11:45:49] 1) stop superset on analytics-tool1003 [11:45:58] 2) backup the db on an-coord1001 [11:46:04] 3) merge + deploy [11:46:12] 4) upgrade the db and check [11:46:37] fdans: sound ok? --^ [11:46:54] elukey perfect [11:49:56] ready to merge [11:50:10] (03CR) 10Elukey: [V: 032 C: 032] Release 0.28.1 [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/479249 (https://phabricator.wikimedia.org/T211605) (owner: 10Elukey) [11:53:19] ImportError: /usr/lib/x86_64-linux-gnu/libmariadbclient.so.18: version `libmysqlclient_18' not found (required by /srv/deployment/analytics/superset/venv/lib/python3.5/site-packages/_mysql.cpython-35m-x86_64-linux-gnu.so) [11:53:28] of course this one didn't pop up in labs [11:58:29] ok fixed, had to upgrade a package [11:59:03] fdans: superset up and running :) [11:59:23] https://superset.wikimedia.org/static/assets/version_info.json looks good [11:59:46] elukey: we got an issue here [11:59:47] https://superset.wikimedia.org/chart/list/ [12:00:43] yeah I noticed that [12:01:56] https://github.com/apache/incubator-superset/issues/6347 [12:01:58] sigh [12:02:49] and it is still open [12:04:05] and the fix is not applicable https://github.com/apache/incubator-superset/pull/6448/commits [12:04:09] since it uses f-strings [12:04:18] only available on python 3.6+ [12:05:24] elukey: yep [12:07:06] is "charts" used frequently? [12:11:08] maybe we can come up with a non fstring solution [12:11:18] they did this [12:11:18] return Markup('{name}').format(**locals()) [12:11:21] return Markup(f'{name}') [12:11:27] we have [12:11:28] TypeError: format() got multiple values for argument 'self' [12:13:47] elukey: pretty sure it’s one of the more used tabs [12:14:37] i mean f strings are pure syntactic sugar [12:15:29] I am wondering if I do something like [12:15:37] url = self.url [12:15:45] return Markup('{name}').format(**locals()) [12:17:59] nope [12:20:32] fdans: ah now it works [12:20:52] ohhh [12:20:52] return Markup('{}').format(self.url, name) [12:21:06] luca you superstar [12:21:29] the main issue is if we'll find these weirdness in other part of the code [12:21:51] probably wise to replace every use of locals() fdans ? [12:22:47] also this is a hotfix that would need a proper deployment [12:22:53] that would be [12:23:04] 1) modify incubator-superset with the patch [12:23:09] 2) rebuild the wheel [12:23:10] etc.. [12:23:13] sigh [12:35:38] fdans: https://github.com/apache/incubator-superset/issues/6347#issuecomment-442178847 [12:35:46] haven't seen this, just applied and it works [12:36:12] they also mention to check markupsafe version [12:37:37] we have ./MarkupSafe-1.1.0-cp35-cp35m-manylinux1_x86_64.whl [12:39:46] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Upgrade Superset to 0.28.1 - https://phabricator.wikimedia.org/T211605 (10elukey) In prod ` (venv) elukey@analytics-tool1003:~$ superset db upgrade Loaded your LOCAL configuration at [/etc/superset/superset_config.py] INFO [alembic.runtime.migration] Con... [12:40:12] elukey: sorry missed your ping earlier [12:40:23] I don't see any issue stopping camus [12:40:46] We'll need to allow for some time before going into safe mode, but shouldn't be too long [12:40:55] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Upgrade Superset to 0.28.1 - https://phabricator.wikimedia.org/T211605 (10elukey) After deploying the Chars panel was broken, fixed manually by https://github.com/apache/incubator-superset/issues/6347#issuecomment-442178847 I am probably just going to reb... [12:48:38] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Bug: can't make a YoY time series chart in Superset - https://phabricator.wikimedia.org/T210687 (10elukey) Me and @fdans deployed 0.28.1 this morning, and we had to apply a hot fix for an outstanding upstream bug (see T211605#4820128 for more juicy info).... [12:57:06] 10Analytics, 10Product-Analytics: Superset Updates - https://phabricator.wikimedia.org/T211706 (10elukey) p:05Triage>03Normal [12:57:54] 10Analytics, 10Product-Analytics: Superset Updates - https://phabricator.wikimedia.org/T211706 (10elukey) [12:59:04] 10Analytics, 10Product-Analytics: As a user of Superset I would like it to be up-to-date so I'm not blocked by bugs that have already been fixed - https://phabricator.wikimedia.org/T211606 (10elukey) @mpopov I added some comments to T211706, let me know what you think about the plan :) [12:59:58] joal: let's plan it for monday if you are ok [13:00:03] no real rush [13:00:33] elukey: as you prefer :) [13:07:50] 10Analytics-EventLogging, 10Analytics-Kanban: [EventLogging Sanitization] Enable older-than-90-day purging of unsanitized EL database (event) in Hive - https://phabricator.wikimedia.org/T209503 (10Miriam) @mforns thanks so much! Would starting executing the script next Monday be ok? Would this leave you enoug... [14:00:59] 10Analytics, 10Dumps-Generation, 10ORES, 10Scoring-platform-team: Produce dump files for ORES scores - https://phabricator.wikimedia.org/T209739 (10ArielGlenn) What sort of help do you need from my end, or is it too early? [14:13:55] milimetric: when you come online, let's talk about the failed job [14:14:20] 10Analytics, 10Operations: setup/install weblog1001/WMF4750 as oxygen replacement - https://phabricator.wikimedia.org/T207760 (10elukey) Created https://phabricator.wikimedia.org/T211883 :) [14:20:30] joal: I should be at Andrew’s in 20-30 minutes, we can talk then [14:45:38] ottomata: o/ [14:45:42] o/ [14:45:43] superset upgraded to 0.28.1 [14:45:47] niiiiice [14:45:51] how'd you end up doing it? [14:46:04] with the custom wheel thing :) [14:46:10] oh wait i have the gerrit change up right now reading [14:46:20] it is useful since we found that 0.28.1 has a nasty bug [14:46:32] that prevents to see the Charts :( [14:46:42] I fixed it manually on an-tool1003 [14:46:50] but I'll need to re-create the wheel with the patch [14:47:37] you didn't have to change anything in superset after you cloned it? [14:47:42] to build it with 3.5? [14:47:55] I had to revert the 3.6 specific patch [14:48:07] I basically did git reset --hard 0.28.1 [14:48:20] git rever #sha-of-the-3.6only-commit [14:48:23] *revert [14:48:25] and then build [14:48:59] I know it is not the cleanest thing in the world but for a couple of months should be fine [14:50:30] silence from Andrew is not good [14:52:14] elukey: he's possibly Hmmmmming alone [14:52:52] joal: if he is doing it while reading my code, it is not a good sign either [14:58:36] joal: I'm going to look at the error and then start cooking [14:58:53] hahahahha [14:59:06] hahah [14:59:12] no i have guests today was welcoming! [14:59:22] we are having a little NYC WMFer cowork day today [14:59:28] elukey: sounds good! [14:59:43] ack! [15:01:36] k, joal so no useful logs in hue, I'm assuming you looked into it. If you want we can hangout while I cook [15:03:44] Hey milimetric [15:03:50] hi [15:04:14] Indeed I found no classical log - problem seems to be related to issues getting intermediary files [15:04:39] milimetric: I suggest we try to run the job as HDFS (even if in your folder), and in production queue [15:07:32] joal: ok, hm, I'll do that as soon as this beef starts to brown [15:08:37] milimetric: take your time - nothing should temper with browning-beef [15:18:26] gone for kids team - will miss standup because of changed school apointment, sending e-scrum - Sorry for tha [15:19:52] hey team! [15:25:51] !log restarted turnilo to clean a deleted test datasource [15:25:53] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:28:20] 10Analytics-EventLogging, 10Analytics-Kanban: [EventLogging Sanitization] Enable older-than-90-day purging of unsanitized EL database (event) in Hive - https://phabricator.wikimedia.org/T209503 (10mforns) @miriam Yes, Monday is good! Thanks :] [15:46:17] job restarted https://hue.wikimedia.org/oozie/list_oozie_workflow/0042326-181112144035577-oozie-oozi-W/?coordinator_job_id=0042325-181112144035577-oozie-oozi-C [15:51:46] 10Analytics-EventLogging, 10Analytics-Kanban: [EventLogging Sanitization] Enable older-than-90-day purging of unsanitized EL database (event) in Hive - https://phabricator.wikimedia.org/T209503 (10Nuria) @mforns Did we updated docs to note that whitelist changes have to go in when schema does? Otherwise data... [15:54:09] elukey, fdans : dashboards in superset have "disaperared" https://superset.wikimedia.org/superset/welcome [15:54:15] *disapeard [15:54:23] elukey: i think we need to rollback [15:54:28] nuria: only in the welcome page tho [15:54:59] nuria: if you go to the dashboard tab they're all there [15:54:59] fdans: wait .. ah i see them [15:55:16] i think the upgrade resets the initial page [15:55:37] fdans: the geoeditors one is not working: https://superset.wikimedia.org/superset/dashboard/9/ [15:55:40] cc elukey [15:56:13] fdans: neither is gfeowiki [16:02:27] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Upgrade Superset to 0.28.1 - https://phabricator.wikimedia.org/T211605 (10Nuria) Adding @Ijon to ticket so he knows this upgrade is happening [16:07:13] fdans (cc elukey ) where will superset traces be? the syslog no longer has them... [16:13:45] 10Analytics, 10Product-Analytics: As a user of Superset I would like it to be up-to-date so I'm not blocked by bugs that have already been fixed - https://phabricator.wikimedia.org/T211606 (10mpopov) 05Open>03declined >>! In T211606#4814078, @elukey wrote: > @mpopov please also keep in mind that things lik... [16:19:20] fdans: ah wait the error on the geowiki dashboards is not on logs cause it is a client side one [16:20:03] nuria: I am in a meeting, checking in a bit [16:20:08] elukey: k [16:22:24] nuria: so the traces should be in syslog but I cannot see anything weird [16:22:31] /var/log/superset/syslog.log [16:22:36] elukey: cause errors are client side, fdans yt? [16:23:35] nuria: sorry, I'm coming back from an errand, will be there soon [16:24:50] fdans: Since two pretty important dashboards are not working we have two choices: 1) recreate them (it might be migration did not worked for these) or 2) rollback to prior version. The upgrade test going forward should include testing all dashboards and most charts - there are not that many- [16:25:33] nuria: so the 0.28.1 upgrade forced a change in the dashboards at the db level, I guess that their procedure is not bullet proof [16:25:46] I checked a couple of them this morning and of course I picked the ones that were working [16:26:22] elukey: i think the issue here is the javascript though it might be totally unrelated to db migration [16:29:48] nuria: let's check which ones are broken, there are 14 so it should be a quick thing [16:30:33] elukey: the geowiki and geoeditors are [16:30:38] elukey: rest seem to be fine [16:31:15] elukey: and they use similar components so i think we are going to be able to fix it, need to attend 30 min meeting will touch base in a bit [16:31:40] ack [16:34:53] as FYI [16:34:54] Superset 0.28 deprecates the previous dashboard layout. While 0.27 offered a migration workflow to users and allowed them to validate and publish their migrated dashboards individually, 0.28 forces the migration of all dashboards through an automated db migration script. We do recommend that you take a backup prior to this migration. [16:38:02] and [16:38:03] Superset 0.27 start to use nested layout for dashboard builder, which is not backward-compatible with earlier dashboard grid data [17:00:59] * fdans so looooonely in standup [17:01:02] a-team [17:01:18] joining! [17:02:29] ping ottomata [17:02:54] ping joal [17:02:57] standddupppp [17:49:11] 10Analytics, 10Analytics-Wikistats: Check wikistats numbers for agreggations for "all-wikipedias" - https://phabricator.wikimedia.org/T189626 (10fdans) a:03fdans [17:49:26] 10Analytics, 10Analytics-Wikistats: Check wikistats numbers for agreggations for "all-wikipedias" - https://phabricator.wikimedia.org/T189626 (10fdans) p:05Unbreak!>03High [17:51:27] 10Analytics: Kerberos stand alone service running in production - https://phabricator.wikimedia.org/T211836 (10fdans) a:03elukey [17:51:50] 10Analytics: Kerberos stand alone service running in production - https://phabricator.wikimedia.org/T211836 (10fdans) p:05Triage>03High [17:52:37] 10Analytics, 10Analytics-Wikimetrics: Sunset Wikimetrics - https://phabricator.wikimedia.org/T211835 (10fdans) p:05Triage>03High [17:54:51] 10Analytics, 10Product-Analytics: [BUG] User agent parsing error for MobileWikiAppSearch table - https://phabricator.wikimedia.org/T211833 (10fdans) a:03Nuria [17:54:53] 10Analytics, 10Analytics-Wikimetrics: Sunset Wikimetrics - https://phabricator.wikimedia.org/T211835 (10Nuria) a:03mforns [17:55:47] 10Analytics, 10Product-Analytics: [BUG] User agent parsing error for MobileWikiAppSearch table - https://phabricator.wikimedia.org/T211833 (10fdans) p:05Triage>03High [17:56:24] 10Analytics, 10Anti-Harassment, 10Product-Analytics: Mediawiki history has no data on IP blocks - https://phabricator.wikimedia.org/T211627 (10fdans) p:05Triage>03Normal [17:57:46] 10Analytics, 10User-Elukey: librdkafka 1.0.0 deprecates functions used by varnishkafka - https://phabricator.wikimedia.org/T210944 (10fdans) p:05Triage>03Low [17:59:26] 10Analytics, 10Patch-For-Review, 10User-Elukey: Varnishkafka error to investigate: Required feature not supported by broker - https://phabricator.wikimedia.org/T210939 (10fdans) p:05Triage>03Low [17:59:53] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Grafana, icinga, prometheus in cloud-analytics project - https://phabricator.wikimedia.org/T211640 (10fdans) p:05High>03Normal [18:01:14] 10Analytics-Kanban, 10Product-Analytics: Superset Updates - https://phabricator.wikimedia.org/T211706 (10fdans) [18:03:39] 10Analytics, 10Developer-Advocacy, 10Product-Analytics, 10Documentation: Develop EventLogging schema for documentation feedback gadget - https://phabricator.wikimedia.org/T211638 (10fdans) If you need any data from this schema to be kept beyond the 90 day window, you need to add it to the whitelist: http... [18:06:13] nuria: hi, tizianop noticed that there's a gap in data here: https://grafana.wikimedia.org/d/000000018/eventlogging-schema?from=1539090159708&to=1539096468963&orgId=1&var-schema=CitationUsage [18:06:26] nuria: it's 10 mins where no data is recorded. Is that a known issue? [18:20:20] bmansurov: is that on data or on graphana? [18:21:55] nuria: afaik we didn't change anything regarding data collection. Maybe eventlogging had a temporary hiccup then? [18:22:06] bmansurov: it is a gap in all schemas [18:22:28] nuria: ok, looks like it's a known issue [18:22:32] bmansurov: which looks like a graphana problem, not a data problem unless you DO see that gap in teh data, makes sense? [18:23:05] nuria: not quite, so grafana is not able to display data for all data sources? [18:23:53] bmansurov: No, graphana also breaks (rarely, but it does), and when it breaks you have no metrics, which does not mean there is no data [18:23:54] tizianop: do you see any data (in hive) that happened in that period? [18:24:05] nuria: ok [18:24:20] bmansurov: makes more sense now? [18:25:43] nuria: I think so. If we don't see data in Hive, then it won't be a grafana issue. [18:26:02] bmansurov: right. [18:26:23] nuria: ok, let's see what tizianop finds. Thanks for your help, nuria! [18:29:18] fdans: want to work together on recreating dashboards on bc? [18:39:36] * elukey afk for a couple of hours [18:45:31] nuria: sorry, I was looking at them outside of irc since the end of the meeting [18:45:51] nuria: basically, the filters box has a bug [18:45:55] fdans: ok, i have a meeting now but shoudl be done in 30 mins [18:45:59] fdans: ya, on js [18:46:21] nuria: the problem is that it's not just these 3 [18:46:39] it's the default state on these three, but all dashboards that have a filters box is affected [18:52:01] Hi team - sorry for earlier :S [18:52:30] 10Analytics, 10Analytics-Wikistats: Wikistats. How about historic data? - https://phabricator.wikimedia.org/T186124 (10Neil_P._Quinn_WMF) Don't we have historic data already? Or is this specific to a particular metric, like [pageviews](https://stats.wikimedia.org/v2/#/all-projects/reading/total-page-views/norm... [18:55:37] nuria: saw your message on pasting the presto-doc content - I suggest copying the doc as-is (or almost) as a new page in wikitech: Analytics/Projects/Data Lake SQL Engine on WMF-Cloud [18:55:41] nuria: ok for you? [18:56:35] something else milimetric - mediawiki-history job has been failing up to now (third ApplicationMaster attemp now running) [18:57:12] milimetric: I'v moved the job to the production queue (was in default) and I'm babysitting closely, trying to better understand if issues happen [18:58:55] joal: Analytics/Projects/Data Lake /SQL Engine on Cloud [18:59:38] nuria: ok for me - wanted to make sure there was no ambiguities on SQL already being available on Cloud :) [19:19:32] 10Quarry: "KeyError: 'request_token'" in /oauth-callback on local instance - https://phabricator.wikimedia.org/T211917 (10Framawiki) p:05Triage>03Low [19:34:04] milimetric: the job issue is a memory one. Kinda unexpected! [19:35:06] manually killing it so that it stops trying with new AMs [19:39:06] 10Analytics, 10Analytics-Wikistats: Wikistats. How about historic data? - https://phabricator.wikimedia.org/T186124 (10Nuria) We do have all history for edits and also pageviews, before 2015 we have pagecounts ("legacy pageviews"): https://stats.wikimedia.org/v2/#/it.wikipedia.org/reading/legacy-page-views/nor... [19:39:53] fdans: free now, want to talk about dashboards? [19:40:08] nuria: yep! [19:40:17] fdans: ok, batcave, let me headset [19:55:30] milimetric: if you're nearby I'd gladly relaunch the testing job with some updates [19:55:44] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Upgrade Superset to 0.28.1 - https://phabricator.wikimedia.org/T211605 (10Nuria) Upgrade run into this issue: https://github.com/apache/incubator-superset/issues/6165 To summarize: the filter component just plain out does not work together with time range... [19:59:48] joal: I'm around [19:59:54] Heya ! [20:00:16] milimetric: Spark doesn't like the join between revisions and comments :( [20:00:20] bah! [20:00:31] I had stopped paying attention to it [20:00:33] just realized it broke [20:00:40] milimetric: 2 things we should update on the job you launch: use production queue, and update memory settings [20:01:09] I thought I shouldn't use production to be nice to the cluster [20:01:15] it doesn't matter if it's slower, no? [20:01:43] I'm worried if we need more memory, these executors are already beefy [20:01:52] milimetric: Here is how I think of updating the memory settings - Max-memory for a container is 42Gb - Spark uses 2 the container memory in 2 slots: main-memory and memory-overhead [20:02:13] We should reduce main-memory and increase the overhead one [20:03:14] We should say: -Dspark_executor_memory=26G -Dspark_executor_memory_overhead=14336 [20:03:18] milimetric: --^ [20:03:27] Hopefully this will help [20:03:32] ok, reduce by 6, increase by 6, I see [20:03:51] milimetric: Using 6 here, but we could go smaller steps [20:05:08] joal: sudo -u hdfs oozie job -run -oozie $OOZIE_URL -config coordinator.properties -Dspark_executor_memory=26G -Dspark_executor_memory_overhead=14336 -Dqueue_name=production [20:05:26] (launching now) [20:05:43] Great milimetric - I'm assuming dates are set in the properties file :) [20:05:52] yes, I always do that just in case [20:06:08] some rogue agent comes and runs oozie jobs from my folder :) [20:06:24] Many thanks for that - Do you mind telling me where I can find this prop. file, in case ? [20:06:36] :D [20:07:06] stat1004:/home/milimetric/refinery/oozie/mediawiki/history/denormalize [20:07:11] ottomata: yt? [20:07:17] great, thanks again milimetric :) [20:07:24] milimetric: will monitor the job [20:07:30] thank you for noticing, I spaced out, sorry [20:07:30] ottomata: we need to do a lovely hotfix for superset [20:08:15] * joal hides at the sound of **lovely hotfix** [20:08:58] milimetric: no problem, I'll be happy when it succeeds :) [20:09:31] milimetric: can you give andrew a nudge if yall are still together? [20:09:45] done [20:09:59] nuria: hoiaaa [20:10:02] what's up? [20:10:27] i got nudged [20:10:31] ottomata: can you explain me how does static data make it to [20:10:40] you can join the batcave party [20:10:44] ottomata: superset.wikimedia.org/static [20:10:55] uhhh [20:11:45] i don't know anything about that url or static data [20:12:13] ottomata: ya, it is part of our build [20:12:19] ottomata: come to bc for a sec [20:12:21] ok [20:17:07] 10Analytics, 10Anti-Harassment (AHT Sprint 35): 👩‍👧 Track how often blocked user attempt to edit - https://phabricator.wikimedia.org/T189724 (10dmaza) [20:20:21] 10Analytics, 10Product-Analytics: [BUG] User agent parsing error for MobileWikiAppSearch table - https://phabricator.wikimedia.org/T211833 (10chelsyx) Update: This is happening in several other tables as well: ` mobilewikiappdailystats mobilewikiappiosfeed mobilewikiappiosloginaction mobilewikiappiosreadinglis... [20:27:30] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Upgrade Superset to 0.28.1 - https://phabricator.wikimedia.org/T211605 (10Nuria) It is becoming pretty clear we are going to need our own fork of superset, creating that now cc @elukey [20:33:17] 10Analytics, 10Analytics-Kanban: Port our "fixes" for python 3.5 to our superset fork https://github.com/wikimedia/incubator-superset - https://phabricator.wikimedia.org/T211932 (10Nuria) p:05Triage→03High [20:36:11] hello hello [20:36:21] elukey: holaa [20:36:35] elukey: kind of later for you but we are in bc [20:36:41] *kind of late [20:57:28] !log stopped superset on analytics-tool1003 for revert to previous version (luca will revert the db backup) [20:57:29] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:07:14] 10Analytics, 10Analytics-Wikistats: Issues with page view map in Wikistats 2 - https://phabricator.wikimedia.org/T200070 (10Jsc39) a:03Jsc39 I want to work on this issue. I am new over here so it would be kind if you could guide me on this [21:11:00] ! superset is back at 0.26.3 [21:11:08] !log superset is back up at version 0.26.3 [21:11:09] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:15:45] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Upgrade Superset to 0.28.1 - https://phabricator.wikimedia.org/T211605 (10Nuria) Upgrade reverted as the new versions are just too unstable, will see if we can backport to our fork the fix for T210687 and we will wait a bit to upgrade tool . Filed some b... [21:16:30] 10Analytics, 10Anti-Harassment (AHT Sprint 35): 👩‍👧 Track how often blocked user attempt to edit - https://phabricator.wikimedia.org/T189724 (10TBolliger) 05Open→03Resolved a:03TBolliger [21:16:34] ;away afi [21:16:46] ouch, that would have been /away etc.. [21:16:48] ahahah [21:16:57] should have been.. [21:17:14] anyway, have a nice evening folks :) [21:18:15] laters elukey! [21:22:19] 10Analytics-Kanban, 10Product-Analytics: Superset Updates - https://phabricator.wikimedia.org/T211706 (10mpopov) Thank you, @Nuria! [21:35:54] (03CR) 10Joal: "Issue noticed while reviewing for performance." (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/476553 (https://phabricator.wikimedia.org/T210543) (owner: 10Milimetric) [21:38:29] (03CR) 10Joal: "Another comment inline." (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/476553 (https://phabricator.wikimedia.org/T210543) (owner: 10Milimetric) [21:42:00] (03CR) 10Milimetric: Join to new actor and comment tables (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/476553 (https://phabricator.wikimedia.org/T210543) (owner: 10Milimetric) [21:44:48] (03CR) 10Joal: "Let's talk about shulffle-keys tomorrow - Sorry for not having noticed the comment at the beginning of table, my super bad :)" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/476553 (https://phabricator.wikimedia.org/T210543) (owner: 10Milimetric) [21:45:30] Gone for tonight - Hopefully the job will succeed (maybe?) - But there still is some improvement to ebmade [21:52:30] joal: FYI! [21:52:33] i think presto is ready [21:52:39] its at least up and ready to play with [21:52:50] so when you start tomorrow have at it! [21:52:55] i haven't put any data in place [21:53:09] but, you can log into ca-coord-1 and run presto --catalog hive [21:53:12] and/or [21:53:13] jsut [21:53:14] hive [21:53:15] to get to hive [21:57:11] 10Analytics, 10Product-Analytics: [BUG] User agent parsing error for MobileWikiAppSearch table - https://phabricator.wikimedia.org/T211833 (10Tbayer) Seems this is also affecting non-iOS EventLogging schemas, e.g. ReadingDepth and NavigationTiming. [22:19:00] (03PS4) 10Milimetric: Add tooltip Vue directive and use it [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/478824 (https://phabricator.wikimedia.org/T177950) [22:44:20] 10Quarry: "KeyError: 'request_token'" in /oauth-callback on local instance - https://phabricator.wikimedia.org/T211917 (10zhuyifei1999) The only reason I can think of'you just have to go back to the main page to be connected' is that the browser requests oauth-callback twice. Is this the case for you? [22:50:54] 10Analytics, 10EventBus, 10Operations, 10Wikidata, and 8 others: WDQS Updater ran into issue and stopped working - https://phabricator.wikimedia.org/T207817 (10CCicalese_WMF) [22:58:15] (03CR) 10Nuria: "Way nicer! Let me try the prod build and the order webpack assigns at build/load just to be super through." (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/478824 (https://phabricator.wikimedia.org/T177950) (owner: 10Milimetric)