[06:37:37] !log Move newly computed snapshot for 2019-05 in place of original one for new checker run to normally succeed [06:37:43] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [06:48:51] joal: bonjour! :D [06:48:56] Good morning :) [06:50:26] All good so far elukey - Will be gone soon for the rest of the morning :) [06:50:51] this is very good :) [06:51:02] anything that I should be aware of? [06:54:01] elukey: I have moved the original snapshot to snapshot=2019-05-original (for history, page_history and user_history_ [06:54:43] And snapshot=2019-05 is the one computed with the new algorithm, for the checker to hopefully be successfull :) [06:54:48] Except from that, all good [06:55:32] 10Analytics, 10Cloud-Services, 10observability, 10User-fgiunchedi: High Prometheus TCP retransmits - https://phabricator.wikimedia.org/T225296 (10elukey) The main issue is that originally I have set up yarn (8141) and hdfs (51010) daemons to bind with ${::ipaddr}:port in puppet, and then we added the IPv6... [06:56:17] super [07:00:51] 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, 10Product-Analytics: page_creation_timestamp not always correct in mediawiki_history - https://phabricator.wikimedia.org/T214490 (10JAllemandou) This is solved in snapshot `2019-05` onward. Some explanation: - The `page_first_edit_timestamp` is th... [07:06:20] 10Analytics: Check home of bawolff - https://phabricator.wikimedia.org/T226955 (10MoritzMuehlenhoff) [07:07:58] 10Analytics, 10Cloud-Services, 10observability, 10Patch-For-Review, 10User-fgiunchedi: High Prometheus TCP retransmits - https://phabricator.wikimedia.org/T225296 (10elukey) Precisely: ` elukey@analytics1031:~$ grep -rni prefer /usr/lib/hadoop/ -B 1 /usr/lib/hadoop/libexec/hadoop-config.sh-245-# Disable... [07:09:07] 10Analytics: Check home of bmansurov - https://phabricator.wikimedia.org/T226956 (10MoritzMuehlenhoff) [07:10:38] 10Quarry, 10MediaWiki-extensions-UserMerge: Query in Toolforge couldn't complete (timeout) - https://phabricator.wikimedia.org/T226949 (10Aklapper) Removing #MediaWiki-Database as your link goes to #mediawiki-extensions-usermerge instead. [07:12:35] 10Analytics: Check home of bmansurov - https://phabricator.wikimedia.org/T226956 (10elukey) Leftovers: ` ====== stat1004 ====== total 0 ls: cannot access '/var/userarchive/bmansurov.tar.bz2': No such file or directory ====== stat1006 ====== total 1700 -rw-r--r-- 1 bmansurov wikidev 34098 Mar 16 2018 dir-che... [07:12:41] 10Analytics: Check home of bmansurov - https://phabricator.wikimedia.org/T226956 (10elukey) a:03leila [07:22:50] 10Quarry, 10MediaWiki-extensions-UserMerge: Query in Toolforge couldn't complete (timeout) - https://phabricator.wikimedia.org/T226949 (10matej_suchanek) Using [[ https://tools.wmflabs.org/sql-optimizer?use=arwiki_p&sql=SELECT+count(actor_name)%0D%0A++++FROM+revision%0D%0A++++INNER+JOIN+page+ON+rev_page+%3D+pa... [07:26:03] 10Analytics, 10Analytics-Kanban, 10Operations, 10vm-requests, 10User-Elukey: Create an-tool1006, a ganeti vm to be used as client for the Hadoop test cluster - https://phabricator.wikimedia.org/T226844 (10elukey) Current status is: ` elukey@ganeti1001:~$ sudo gnt-group list Group Nodes Instances AllocP... [07:29:59] 10Analytics, 10Operations: Reduce memory allocation for kafkamon instances - https://phabricator.wikimedia.org/T224988 (10elukey) I would go down to 4G with (on ganeti1001): ` sudo gnt-instance modify -B memory=4g kafkamon1001.eqiad.wmnet ` Same thing for the codfw instance. From grafana it seems that we co... [07:30:21] 10Analytics, 10Analytics-Kanban, 10Operations: Reduce memory allocation for kafkamon instances - https://phabricator.wikimedia.org/T224988 (10elukey) a:03elukey [09:47:46] 10Analytics, 10Analytics-Kanban, 10Operations: Reduce memory allocation for kafkamon instances - https://phabricator.wikimedia.org/T224988 (10akosiaris) >>! In T224988#5295172, @elukey wrote: > I would go down to 4G with (on ganeti1001): > > ` > sudo gnt-instance modify -B memory=4g kafkamon1001.eqiad.wmne... [09:53:21] 10Analytics, 10Analytics-Kanban, 10Operations, 10vm-requests, 10User-Elukey: Create an-tool1006, a ganeti vm to be used as client for the Hadoop test cluster - https://phabricator.wikimedia.org/T226844 (10akosiaris) >>! In T226844#5295161, @elukey wrote: > Current status is: > > ` > elukey@ganeti1001:~$... [10:17:55] 10Analytics, 10Cloud-Services, 10observability, 10User-fgiunchedi: High Prometheus TCP retransmits - https://phabricator.wikimedia.org/T225296 (10elukey) All the Hadoop testing cluster is running with IPv6 addresses bound, it looks good (didn't see anything failing so far). I'll wait a couple of days and t... [10:18:07] 10Analytics, 10Cloud-Services, 10observability, 10User-Elukey, 10User-fgiunchedi: High Prometheus TCP retransmits - https://phabricator.wikimedia.org/T225296 (10elukey) [10:23:42] 10Analytics, 10Analytics-Kanban: Disable Hive querying in Superset - https://phabricator.wikimedia.org/T223919 (10elukey) The hadoop masters have been rebooted, so the change is now in effect. Since proper auth is not in place yet, superset needs to be manually cleaned up. I tried to delete the Hive database,... [10:25:50] 10Analytics, 10Analytics-Kanban: Disable Hive querying in Superset - https://phabricator.wikimedia.org/T223919 (10elukey) Found a way, removed tables and finally the Hive database from superset. [10:25:54] 10Analytics, 10Analytics-Kanban: Disable Hive querying in Superset - https://phabricator.wikimedia.org/T223919 (10elukey) [10:26:11] !log removed Hive tables and Database from Superset - T223919 [10:26:14] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:26:14] T223919: Disable Hive querying in Superset - https://phabricator.wikimedia.org/T223919 [10:39:27] * elukey lunch! [11:36:26] 10Quarry, 10MediaWiki-extensions-UserMerge: Query in Toolforge couldn't complete (timeout) - https://phabricator.wikimedia.org/T226949 (10Jar) @matej_suchanek same problem when I use the `revision_userindex` table. [11:42:52] 10Quarry, 10Regression: Query in Toolforge couldn't complete (timeout) - https://phabricator.wikimedia.org/T226949 (10matej_suchanek) [13:33:49] 10Analytics, 10Research: Check home of bmansurov - https://phabricator.wikimedia.org/T226956 (10leila) [13:34:45] 10Analytics, 10Research: Check home of bmansurov - https://phabricator.wikimedia.org/T226956 (10leila) On it. Working with Baha to figure out which ones to keep and which ones to drop. (I'm off some of this week and a day next week. I'll be slow.) [13:43:42] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Add UTC 'Z' suffix to webrequest `dt` field. - https://phabricator.wikimedia.org/T217040 (10Ottomata) [13:52:08] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Product-Analytics, and 3 others: Use Z UTC suffix in EventBus emitted events rather than +00:00 - https://phabricator.wikimedia.org/T217041 (10Ottomata) [13:52:10] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Core Platform Team Backlog (Watching / External), and 2 others: Modern Event Platform: Stream Intake Service: Migrate Mediawiki Eventbus events to eventgate-main - https://phabricator.wikimedia.org/T211248 (10Ottomata) [13:52:49] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Product-Analytics, and 3 others: Use Z UTC suffix in EventBus emitted events rather than +00:00 - https://phabricator.wikimedia.org/T217041 (10Ottomata) As each event is migrated to the new Event Platform format in T211248, the timestamps will use the 'Z' suff... [14:12:43] 10Analytics: Research wether we can throttle the number of files created by a job so namenode does not get overwhelmed - https://phabricator.wikimedia.org/T220126 (10elukey) [14:12:45] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Check if HDFS offers a way to prevent/limit/throttle users to overwhelm the HDFS Namenode - https://phabricator.wikimedia.org/T220702 (10elukey) [14:17:14] 10Analytics, 10Product-Analytics: Bug: Superset asking for my credentials on every page load - https://phabricator.wikimedia.org/T224159 (10elukey) @JKatzWMF, @kzimmerman - Still happening? I am wondering if a specific version of Chrome is causing this.. Kate's version is `Chrome Version 75.0.3770.90 (Official... [14:17:32] 10Analytics, 10Analytics-EventLogging, 10DBA, 10Operations, 10ops-eqiad: db1107 (eventlogging db master) possibly memory issues - https://phabricator.wikimedia.org/T222050 (10elukey) ping :) [14:18:55] 10Analytics, 10Analytics-SWAP: Upgrade R in SWAP notebooks to 3.4+ - https://phabricator.wikimedia.org/T222933 (10elukey) [14:18:57] 10Analytics, 10Product-Analytics: Update R from 3.3.3 to 3.5.3 on stat and notebook machines - https://phabricator.wikimedia.org/T220542 (10elukey) [14:24:58] 10Analytics, 10Product-Analytics: Update R from 3.3.3 to 3.5.3 on stat and notebook machines - https://phabricator.wikimedia.org/T220542 (10elukey) @Ottomata do you think that this could be doable without going to buster? I am a bit ignorant about cran and r packages, but I can check later on during the week i... [14:26:26] 10Analytics, 10Product-Analytics: Update R from 3.3.3 to 3.5.3 on stat and notebook machines - https://phabricator.wikimedia.org/T220542 (10Ottomata) Not sure, but we are also waiting for buster to upgrade Spark. When I asked Moritz before, he said Buster would be ready in a monthish time. [14:27:36] 10Analytics, 10Product-Analytics: Update R from 3.3.3 to 3.5.3 on stat and notebook machines - https://phabricator.wikimedia.org/T220542 (10elukey) Yep I agree, but moving the notebooks/etc.. to Buster might be a long project (say if we need to rebuild packages, change configs for new versions, etc..). Unblock... [14:27:54] 10Analytics, 10Product-Analytics: Update R from 3.3.3 to 3.5.3 on stat and notebook machines - https://phabricator.wikimedia.org/T220542 (10MoritzMuehlenhoff) Buster will be released next Saturday [14:29:05] 10Analytics, 10User-Elukey: Show IPs matching a list of IP subnets in Webrequest data - https://phabricator.wikimedia.org/T220639 (10elukey) > How do we run this with a venv so that we can include Pytricia? Ideally if we had a deb package for this library we could deploy it on all the worker nodes and use it... [14:31:20] 10Analytics, 10Operations, 10Wikimedia-Logstash, 10service-runner, and 2 others: Move AQS logging to new logging pipeline - https://phabricator.wikimedia.org/T219928 (10elukey) @Nuria, do you think that we could work on this during the next couple of months? Seems to be an easy enough change to be ready in... [14:32:42] 10Analytics, 10Patch-For-Review: Review Bacula home backups set for stat100[56] - https://phabricator.wikimedia.org/T201165 (10elukey) @Ottomata let's decide what policy to adopt during our next ops sync! (try to get backups again or just close the task) [14:50:34] 10Analytics, 10Operations, 10Wikimedia-Logstash, 10service-runner, and 2 others: Move AQS logging to new logging pipeline - https://phabricator.wikimedia.org/T219928 (10Nuria) Let's plan this for this quarter then? (q2?) [14:55:49] (03CR) 10Fdans: "I think that behavior is normal considering that the "all" range in legacy pageviews is outside of the range of the total pageviews range," [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/519016 (https://phabricator.wikimedia.org/T226486) (owner: 10Fdans) [15:05:15] (03PS4) 10Fdans: Add zeroes to truncated values and UI about truncation [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/519382 (https://phabricator.wikimedia.org/T220098) [15:38:54] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Make timers that delete data use the new deletion script - https://phabricator.wikimedia.org/T226862 (10fdans) p:05Triage→03High [15:40:17] 10Analytics, 10Analytics-Kanban, 10Operations, 10vm-requests, 10User-Elukey: Create an-tool1006, a ganeti vm to be used as client for the Hadoop test cluster - https://phabricator.wikimedia.org/T226844 (10fdans) p:05Triage→03High [15:41:20] 10Analytics, 10Analytics-Kanban: Fix Hive partition thresholding in refinery-drop-older-than - https://phabricator.wikimedia.org/T226835 (10fdans) 05Open→03Resolved [15:41:42] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Allow all Analytics tools to work with Kerberos auth - https://phabricator.wikimedia.org/T226698 (10fdans) p:05Triage→03High [15:43:59] 10Analytics, 10EventBus, 10Core Platform Team Backlog (Watching / External), 10Services (watching): Factor out eventgate-wikimedia factory into its own gerrit repo and use it for deployment pipeline - https://phabricator.wikimedia.org/T226668 (10fdans) p:05Triage→03Normal [15:44:06] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Core Platform Team Backlog (Watching / External), 10Services (watching): Factor out eventgate-wikimedia factory into its own gerrit repo and use it for deployment pipeline - https://phabricator.wikimedia.org/T226668 (10fdans) [15:44:45] 10Analytics, 10EventBus, 10Core Platform Team Backlog (Watching / External), 10Services (later): Modern Event Platform: Stream Intake Service: Migrate change-prop events to new (EventGate) style schemas - https://phabricator.wikimedia.org/T226522 (10fdans) p:05Triage→03Normal [15:45:42] 10Analytics: Map doesn't redraw when returning from table view - https://phabricator.wikimedia.org/T226514 (10fdans) p:05Triage→03High [15:46:58] 10Analytics, 10Cleanup, 10Operations: Archive cdh puppet submodule - https://phabricator.wikimedia.org/T226474 (10fdans) p:05Triage→03Normal [15:47:10] 10Analytics, 10Analytics-Kanban, 10Cleanup, 10Operations: Archive cdh puppet submodule - https://phabricator.wikimedia.org/T226474 (10fdans) [15:47:29] 10Analytics, 10Operations, 10Patch-For-Review, 10Security, and 2 others: Eventstreams in codfw down for several hours due to kafka2001 -> kafka-main2001 swap - https://phabricator.wikimedia.org/T226808 (10Ottomata) [15:48:02] 10Analytics: User knissen can't access Superset - https://phabricator.wikimedia.org/T226431 (10fdans) a:05kai.nissen→03Nuria [15:51:02] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Update the Camus checker to be able to authenticate via Kerberos - https://phabricator.wikimedia.org/T226232 (10fdans) 05Open→03Resolved [15:51:04] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Run critical Analytics Hadoop jobs and make sure that they work with the new auth settings. - https://phabricator.wikimedia.org/T212259 (10fdans) [16:17:11] 10Analytics, 10Product-Analytics, 10Growth-Team (Current Sprint): Homepage: specify purging strategy - https://phabricator.wikimedia.org/T219252 (10nettrom_WMF) [16:17:15] 10Analytics, 10Product-Analytics, 10Growth-Team (Current Sprint): Homepage: add schemas to EventLogging whitelist - https://phabricator.wikimedia.org/T225471 (10nettrom_WMF) 05Open→03Resolved Have confirmed that the data is now flowing into the Data Lake and appears to be correct. Closing this as resolved. [17:50:26] PROBLEM - Check the last execution of refinery-sqoop-whole-mediawiki on an-coord1001 is CRITICAL: CRITICAL: Status of the systemd unit refinery-sqoop-whole-mediawiki [17:59:54] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Core Platform Team Backlog (Watching / External), 10Services (watching): Modern Event Platform: Schema Registry - https://phabricator.wikimedia.org/T201063 (10Ottomata) [18:00:01] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Core Platform Team Backlog (Watching / External), 10Services (watching): Modern Event Platform: Schema Registry - https://phabricator.wikimedia.org/T201063 (10Ottomata) [18:05:33] * elukey off! [18:15:05] Nettrom: generating some events on arwiki for editorjourney now [18:15:26] kostajh: monitoring! [18:20:28] Nettrom: seems OK so far, yeah? [18:20:49] Is there any way to know top contributors of all time on a wiki? I can get it on old stats.wikimedia.org but how to get on new stats2 [18:21:13] kostajh: they are flowing in, yes! data looks fine to me [18:21:38] Nettrom: cool, we're deploying! [18:23:00] kostajh: verified that obfuscation was in place where expected as well, btw [18:23:32] Nettrom: thanks [18:27:44] gyan: in wikistats2 that data is only available per month: see https://wikitech.wikimedia.org/wiki/Analytics/AQS/Wikistats_2#Top_Editors [18:29:54] Thanks is there any other way to know the stats about top contributor of all time? Our wiki just hit 15k and we want to give a shoutout on social media. Is quarry is an option? [18:31:22] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Core Platform Team (Modern Event Platform (TEC2)), and 3 others: Modern Event Platform (TEC2) - https://phabricator.wikimedia.org/T185233 (10Ottomata) [18:31:41] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Core Platform Team (Modern Event Platform (TEC2)), and 3 others: Modern Event Platform (TEC2) - https://phabricator.wikimedia.org/T185233 (10Ottomata) [18:46:07] gyan: for all time? it probably depends on size of wiki, i would request data per month and do a tally [18:46:14] gyan: what wiki is this one? [18:46:29] orwiki [19:01:17] gyan: I see, best you can do is do a program that request from api all months since 2004 when wiki was created https://stats.wikimedia.org/v2/#/or.wikipedia.org/contributing/editors/normal|line|all|~total|monthly [19:02:24] Thank you. [19:31:52] hi yall [19:32:09] I'm just here to move along the sqoop jobs [19:32:52] I am going to manually put up some _SUCCESS flags to ignore hiwikisource and I have to figure out how to restore and continue with the systemd timer that runs two jobs [19:32:58] any help with that last part is appreciated ^ [19:33:48] so, we have refinery-sqoop-whole-mediawiki that runs two sqoops, and it's in critical [19:34:27] if I manually fix one of the sqoops, how do I tell it to keep going with the second one and not be in a critical state any more? I'll read the docs, but if anyone like ottomata could help, that'd be useful [19:35:15] milimetric: i'm around can help, not familiar with this part of the system but we can figure it out [19:35:22] i'm deploying some eventstreams changes now [19:35:27] can be with you after that. [19:36:25] ok, no prob, I'll read up [19:36:28] thanks [19:46:25] ok, success flags added correctly, now trying to figure out this systemd timer: https://github.com/wikimedia/puppet/blob/7269a0ce1fc089aeee08916db030130ef5b9fc4f/modules/profile/manifests/analytics/refinery/job/sqoop_mediawiki.pp#L62 [19:52:36] dan [19:52:39] let's do that now [19:52:45] i thhink i'll deploy eventstreams tomorrow [19:52:46] somethign is weird [19:52:53] ottomata: ok, I'll jump in cave [20:01:53] Hi folks :) [20:02:08] Just seen the sqoop errors [20:02:21] joal: all fixed, go away! [20:02:22] :) [20:02:26] Ok :) [20:03:10] milimetric: let's look at the systemd timer? [20:03:19] milimetric: or did you alredy do that with ottomata [20:03:24] nuria: I can sync up since I may miss standup tomorrow [20:03:40] milimetric: sure [20:03:41] nuria: (in cave) [20:06:22] RECOVERY - Check the last execution of refinery-sqoop-whole-mediawiki on an-coord1001 is OK: OK: Status of the systemd unit refinery-sqoop-whole-mediawiki [20:06:36] 10Analytics, 10DBA: hi.wikisource added to labs replicas? - https://phabricator.wikimedia.org/T227030 (10Nuria) [20:11:43] 10Analytics, 10DBA: hi.wikisource added to labs replicas? - https://phabricator.wikimedia.org/T227030 (10Nuria) Also seems that it is not replicated to analytics production replicas? [20:13:05] (03PS1) 10Milimetric: Remove hiwikisource because it does not exist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/520077 [20:13:18] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Remove hiwikisource because it does not exist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/520077 (owner: 10Milimetric) [20:13:34] (03CR) 10Nuria: [C: 03+2] Remove hiwikisource because it does not exist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/520077 (owner: 10Milimetric) [20:19:35] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Issues with page deleted dates on data lake - https://phabricator.wikimedia.org/T190434 (10JAllemandou) Improved greatly by the last page-history reconstruction refactor: ` spark.sql(""" WITH rev_deleted AS ( SELECT wiki_db as deleted_rev_wiki,... [20:19:55] !log syncing to hdfs on minor refinery deploy to remove hiwikisource from sqoop lists [20:19:57] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:29:01] !log removed old refinery deploy caches from an-coord1001 to free up disk space [20:29:03] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:32:42] 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, 10Product-Analytics: Many revision events in mediawiki_history have missing page and namespace information - https://phabricator.wikimedia.org/T221338 (10JAllemandou) This is solved from snapshot `2019-05` onward thanks to the rebuild of the page-h... [20:32:50] 10Analytics, 10Operations, 10Patch-For-Review, 10Security, and 2 others: Eventstreams in codfw down for several hours due to kafka2001 -> kafka-main2001 swap - https://phabricator.wikimedia.org/T226808 (10Ottomata) Ok, all patches ready to go. Deployed in beta and looks good there. It is near the end of... [20:33:50] 10Analytics, 10Performance-Team: Plan navtiming data release - https://phabricator.wikimedia.org/T214925 (10Krinkle) [20:34:00] 10Analytics, 10Performance-Team: Plan navtiming data release - https://phabricator.wikimedia.org/T214925 (10Krinkle) [20:34:07] 10Analytics, 10Performance-Team: Release performance data on a regular schedule - https://phabricator.wikimedia.org/T205342 (10Krinkle) a:03Gilles [20:34:21] 10Analytics, 10Performance-Team: Release performance data on a regular schedule - https://phabricator.wikimedia.org/T205342 (10Krinkle) 05Open→03Stalled p:05Triage→03High [20:35:34] ottomata: looks like some weird SCAP deploy error on an-coord1001, I tried deploying again and it didn't work [20:35:42] but maybe don't do anything 'cause my job is running on a screen there [20:35:52] should be done in a couple hours [20:39:49] 10Analytics, 10Analytics-Kanban: Decide: start_timestamp for mediawiki history - https://phabricator.wikimedia.org/T220507 (10JAllemandou) We have not implemented the proposal defined here for page-create event timestamp definition. I let @Milimetric explain (either here or in sync-up meeting, might be easier... [20:39:57] 10Analytics, 10Analytics-Kanban: Mediawiki-history release - Snapshot 2019-06 - https://phabricator.wikimedia.org/T221825 (10JAllemandou) [20:40:48] 10Analytics, 10Analytics-Kanban: Decide: start_timestamp for mediawiki history - https://phabricator.wikimedia.org/T220507 (10Milimetric) Quick note that we tried to do what we proposed here but it complicated other parts of the data too much. So we reverted to, for now, the following: 1. use the first revis... [21:03:19] milimetric: were you able to deploy? ah i see that ottomata cleaned up files srv/deployment/analytics/refinery-cache/revs/ [21:05:12] nuria: yeah, I ran the job regardless, it wasn’t affected, wiki list was copied ok [21:13:43] (03PS6) 10Nuria: Change number formating to show less decimal places [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/519036 (https://phabricator.wikimedia.org/T200070) (owner: 10Fdans) [21:14:09] milimetric: can you take a look at this change regarding formatting? https://gerrit.wikimedia.org/r/#/c/analytics/wikistats2/+/519036 [21:14:37] milimetric: i talked about it with marcel but did not pushed on friday [21:15:46] https://usercontent.irccloud-cdn.com/file/SvJ6jNOW/Screen%20Shot%202019-07-01%20at%202.15.32%20PM.png [21:15:49] before [21:15:57] https://usercontent.irccloud-cdn.com/file/jeBWkkuS/Screen%20Shot%202019-07-01%20at%202.15.25%20PM.png [21:16:07] after [21:18:03] nuria: makes sense in general, but it's weird that 1.07B rounds to 1.1B, whereas 12.78B rounds to 13B, that's a much bigger rounding [21:18:09] I'd be ok if that said 12.8B [21:18:40] otherwise I support this kind of rounding, though maybe it'd be even better if it rounded down to the nearest 1 decimal place [21:18:46] so 12.78 -> 12.7 [21:19:20] that wouldn't cause any trouble, whereas I would imagine 12.51B getting rounded to 13B would be kind of a big lie [21:19:45] ya numeral is not the best for rounding , it uses Math.round (unless you give it a rounding function) but rounds sometimes 12 separated from 78 [21:21:15] milimetric: can do this [21:23:18] wait, where do you see the 12.78? [21:23:35] milimetric: ah sorry, ya bottom of panel [21:23:43] yep [21:30:57] milimetric: if it was 12.51 it will probably say 13B if we format it with 0a with 0.0a it will say 12.5 , *i think* it applies the format and after the rounding [21:31:06] milimetric: which is counter intuitive [21:32:03] yeah, 0.0a is what we used before I think [21:38:07] milimetric: Math.round(12.7) [21:38:12] milimetric: sorry [21:39:30] milimetric: so 12.51 with the precision we are writing it (2 significant digits) will indeed become 13 [21:50:37] 10Analytics, 10Analytics-Wikistats: 'All" time range does not transfer well across metrics - https://phabricator.wikimedia.org/T227038 (10Nuria) [21:51:49] (03CR) 10Nuria: [C: 03+2] "I agree that Marcel is right behaviour could be improved and have filed ticket on this regard: https://phabricator.wikimedia.org/T227038 I" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/519016 (https://phabricator.wikimedia.org/T226486) (owner: 10Fdans) [21:52:21] (03CR) 10Nuria: [V: 03+2 C: 03+2] Create "all" time ranges based on the metric config [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/519016 (https://phabricator.wikimedia.org/T226486) (owner: 10Fdans) [21:53:10] milimetric: let me know if you feel strongly about the rounding and think it cannot be merged that way. [21:53:50] milimetric: i can always change math.round to math.floor so it will always be an underestimate [22:01:46] 10Analytics: User knissen can't access Superset - https://phabricator.wikimedia.org/T226431 (10Nuria) Please try again, I think a piece of config was missing on your user. [22:19:18] 10Analytics, 10DBA: hi.wikisource added to labs replicas? - https://phabricator.wikimedia.org/T227030 (10Reedy) It hasn't actually been properly created... It's not public. ie you can't get on https://hi.wikisource.org See {T218155} and {T212881} [22:34:57] 10Analytics, 10Product-Analytics: Hash all pageTokens or temporary identifiers from the EL Sanitization white-list as needed for iOS - https://phabricator.wikimedia.org/T226849 (10chelsyx) p:05Triage→03Normal [22:35:47] (03PS1) 10Chelsyx: Hash tokens from the EL Sanitization white-list for iOS app [analytics/refinery] - 10https://gerrit.wikimedia.org/r/520134 (https://phabricator.wikimedia.org/T226849) [22:41:29] 10Analytics, 10Product-Analytics, 10Patch-For-Review: Hash all pageTokens or temporary identifiers from the EL Sanitization white-list as needed for iOS - https://phabricator.wikimedia.org/T226849 (10chelsyx) I hash the tokens for the following EL schemas used by the iOS app: ` MobileWikiAppEdit MobileWikiAp... [22:44:59] 10Analytics: [BUG] Logging error of MobileWikiAppDailyStats for the iOS app - https://phabricator.wikimedia.org/T226219 (10Nuria) @chelseyx: maria db storage will accept non backwards compatible changes in schemas but not hive, it is not possible to support non backwards compatible schemas in storage backed up... [22:52:05] 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Many small wikis missing from mediawiki_history dataset - https://phabricator.wikimedia.org/T220456 (10Nuria) closing as after scooping looks like the only wiki failed was hiwikisource so rest existed. [22:52:24] 10Analytics-Kanban, 10Product-Analytics: Address data quality issues in the mediawiki_history dataset - https://phabricator.wikimedia.org/T204953 (10Nuria) [22:52:24] 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Many small wikis missing from mediawiki_history dataset - https://phabricator.wikimedia.org/T220456 (10Nuria) 05Open→03Resolved [22:52:30] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban: Fix duplicate puppet class profile::hadoop::users vs profile::analytics::cluster::users - https://phabricator.wikimedia.org/T225464 (10Nuria) 05Open→03Resolved [22:53:01] 10Analytics, 10Analytics-Kanban: Finish actor migration in refinery sqoop and refinery source - https://phabricator.wikimedia.org/T224134 (10Nuria) 05Open→03Resolved [22:53:29] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Product-Analytics, and 3 others: Use Z UTC suffix in EventBus emitted events rather than +00:00 - https://phabricator.wikimedia.org/T217041 (10Nuria) 05Open→03Resolved [22:53:32] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Core Platform Team Backlog (Watching / External), and 3 others: Modern Event Platform: Stream Intake Service: Migrate Mediawiki Eventbus events to eventgate-main - https://phabricator.wikimedia.org/T211248 (10Nuria) [22:54:09] 10Analytics, 10Analytics-Kanban: Disable Hive querying in Superset - https://phabricator.wikimedia.org/T223919 (10Nuria) 05Open→03Resolved [23:02:30] 10Analytics: [BUG] Logging error of MobileWikiAppDailyStats for the iOS app - https://phabricator.wikimedia.org/T226219 (10Nuria) Now, I would expect events that do not validate (to the latest version of schema) to be logged in on eventterror, right? pinging @Ottomata [23:02:39] 10Analytics: [BUG] Logging error of MobileWikiAppDailyStats for the iOS app - https://phabricator.wikimedia.org/T226219 (10Nuria) a:05Nuria→03Ottomata [23:06:10] 10Analytics, 10Product-Analytics: Bug: Superset asking for my credentials on every page load - https://phabricator.wikimedia.org/T224159 (10JKatzWMF) @elukey Still happening. This is me: Version 75.0.3770.100 (Official Build) (64-bit) I think the morale here is that I am 00.0.0000.010 cooler than Kate. I... [23:08:23] 10Analytics, 10Product-Analytics: Bug: Superset asking for my credentials on every page load - https://phabricator.wikimedia.org/T224159 (10Nuria) And versions of mac OS?