[00:00:45] (03PS1) 10Alaa Sarhan: Pass wbqsHost to service instance [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/521379 [00:01:21] (03CR) 10jerkins-bot: [V: 04-1] Pass wbqsHost to service instance [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/521379 (owner: 10Alaa Sarhan) [00:06:15] (03PS2) 10Alaa Sarhan: Pass wbqsHost to service instance [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/521379 [00:07:45] (03CR) 10Alaa Sarhan: [C: 03+1] "follow up patch https://gerrit.wikimedia.org/r/c/analytics/wmde/scripts/+/521379" (031 comment) [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/520903 (https://phabricator.wikimedia.org/T218710) (owner: 10Ladsgroup) [03:24:45] 10Analytics, 10Pageviews-API, 10Tool-Pageviews: 429 Too Many Requests hit despite throttling to 100 req/sec - https://phabricator.wikimedia.org/T219857 (10MusikAnimal) [03:25:51] 10Analytics, 10Pageviews-API, 10Tool-Pageviews: 429 Too Many Requests hit despite throttling to 100 req/sec - https://phabricator.wikimedia.org/T219857 (10MusikAnimal) [06:12:49] 10Analytics, 10Operations, 10Patch-For-Review, 10User-Elukey: Import AMD rocm packages in wikimedia-buster - https://phabricator.wikimedia.org/T224723 (10elukey) Better now! ` root@install1002:/srv/wikimedia# reprepro --noskipold --component thirdparty/amd-rocm checkupdate buster-wikimedia Calculating pac... [06:23:59] 10Analytics, 10Analytics-Kanban, 10ExternalGuidance, 10Product-Analytics: [Bug] `init` and `mtinfo` event counts drop drastically since June 17 2019 - https://phabricator.wikimedia.org/T227150 (10chelsyx) > @chelsyx Besides translate.googleusercontent.com is there any other third party domain sending us da... [06:53:06] 10Analytics, 10Operations, 10Patch-For-Review, 10User-Elukey: Import AMD rocm packages in wikimedia-buster - https://phabricator.wikimedia.org/T224723 (10elukey) New list: ` root@install1002:/srv/wikimedia# reprepro --noskipold --component thirdparty/amd-rocm checkupdate buster-wikimedia Calculating packa... [08:15:44] 10Analytics: Jan Dittrich would like to have access to superset - https://phabricator.wikimedia.org/T227093 (10Jan_Dittrich) [08:16:11] 10Analytics: Jan Dittrich would like to have access to superset - https://phabricator.wikimedia.org/T227093 (10Jan_Dittrich) @Nuria sure – ldap: WMDE-jand [09:54:04] 10Analytics, 10Operations, 10Patch-For-Review, 10User-Elukey: Import AMD rocm packages in wikimedia-buster - https://phabricator.wikimedia.org/T224723 (10elukey) Now the annoying part: ` elukey@stat1005:~$ apt-cache show hsa-rocr-dev Package: hsa-rocr-dev Status: install ok installed Priority: optional Se... [10:42:13] elukey: whats the safest way to restart the aqs. is it safe to just ` systemctl restart aqs.service` ? [10:44:47] yeah, you can depool, systemctl restart aqs.service, repool [10:45:01] jbond42: hello :) I usually do a very paranoid -m async 'depool 'sleep 5' 'systemctl restart aqs' 'sleep 5' 'pool' -b 1 -s 10 with cumin [10:45:23] thanks both ill add that to the restarts wiki [11:23:06] (03PS3) 10Ladsgroup: Use config for wdqs host name [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/520903 (https://phabricator.wikimedia.org/T218710) [11:23:11] (03CR) 10Ladsgroup: Use config for wdqs host name (031 comment) [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/520903 (https://phabricator.wikimedia.org/T218710) (owner: 10Ladsgroup) [11:53:01] amd rocm packages deployed via puppet on stat1005! [11:53:05] \o/ [12:01:52] * elukey lunch! [13:26:43] !log enable base::firewall on stat1007 [13:26:45] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:17:40] ottomata: o/ [14:17:42] I created https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/AMD_GPU [14:17:49] let me know if it makes sense or not [14:18:46] nice elukey! [14:18:49] 10Analytics, 10Operations, 10Patch-For-Review, 10User-Elukey: Import AMD rocm packages in wikimedia-buster - https://phabricator.wikimedia.org/T224723 (10elukey) Added documentation in https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/AMD_GPU [14:19:51] ottomata: it is still a bit cumbersome but I hope that upstream will ease the process :( [14:20:12] aye [14:20:23] also, if you are ok I'd enable the firewall on the notebooks [14:20:54] ah snap there are active spark sessions [14:21:45] I might need to do something like [14:21:53] 1) add the spark driver port 12000 [14:22:00] wait for a week [14:22:02] 2) add firewall [14:22:10] otherwise I think I'll impact people [14:22:27] hm aye [14:22:36] oh because the active sessions are out of the port rnge [14:22:37] right [14:23:08] yeah [14:23:19] I think that ferm will not accept traffic from them [14:27:18] aye [14:46:23] elukey: i think ferm will only block new incoming connections, but ya if e.g. a new exectutor is launched, it will attempt to make a new connection and fail [14:47:54] !log moved all mediawiki_page_* event tables to schema aware refine job [14:47:55] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:50:19] ottomata: yep yep Moritz confirmed, but since there is no rush let's split :) [14:54:47] cool [15:06:46] 10Analytics: Jan Dittrich would like to have access to superset - https://phabricator.wikimedia.org/T227093 (10Nuria) Please try to log in http://superset.wikimedia.org [15:15:12] 10Analytics, 10Analytics-Kanban, 10ExternalGuidance, 10Product-Analytics: [Bug] `init` and `mtinfo` event counts drop drastically since June 17 2019 - https://phabricator.wikimedia.org/T227150 (10Nuria) @chelsey: we will be needing to whitelist external domains cause most EL traffic that comes from 3rd pa... [15:27:17] elukey: the sessions can be restarted very easily [15:27:28] elukey: just with areload of your notebook basically [15:28:03] nuria: morning! I'd prefer to let the users do that, just to avoid impacting their work.. I'll send some emails [15:28:23] elukey: ok. [15:29:21] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats2: Values in map view show unnecessary decimal digits - https://phabricator.wikimedia.org/T200070 (10Nuria) 05Open→03Resolved [15:29:37] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Fix status overlay for dates out of bounds - https://phabricator.wikimedia.org/T226402 (10Nuria) 05Open→03Resolved [15:29:54] 10Analytics, 10Analytics-Kanban: Mediawiki-history release - Snapshot 2019-06 - https://phabricator.wikimedia.org/T221825 (10Nuria) 05Open→03Resolved [15:30:06] 10Analytics, 10Analytics-Kanban: "All" time range selection should be aware of the metric's available time range - https://phabricator.wikimedia.org/T226486 (10Nuria) 05Open→03Resolved [15:31:04] 10Analytics, 10Analytics-Kanban: Exclude doc.wikimedia.org from pageview definition - https://phabricator.wikimedia.org/T225792 (10Nuria) 05Open→03Resolved [15:40:03] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Issues with page deleted dates on data lake - https://phabricator.wikimedia.org/T190434 (10Nuria) 05Open→03Resolved [15:40:04] 10Analytics, 10Analytics-Kanban: Mediawiki-history release - Snapshot 2019-06 - https://phabricator.wikimedia.org/T221825 (10Nuria) [15:40:06] 10Analytics-Kanban, 10Product-Analytics: Address data quality issues in the mediawiki_history dataset - https://phabricator.wikimedia.org/T204953 (10Nuria) [15:43:27] hey yall, I’m not feeling well, will skip meetings except my 1/1 with nuria, trying to get some rest [15:43:29] 10Analytics, 10Analytics-Kanban, 10ExternalGuidance, 10Product-Analytics: [Bug] `init` and `mtinfo` event counts drop drastically since June 17 2019 - https://phabricator.wikimedia.org/T227150 (10dr0ptp4kt) @chelsyx I forget some of the details, but I think it's okay to allow the multiple domains be the re... [15:44:02] 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, 10Product-Analytics: page_creation_timestamp not always correct in mediawiki_history - https://phabricator.wikimedia.org/T214490 (10Nuria) Ping @Milimetric do we need to update docs in wikitech after last refactor? https://wikitech.wikimedia.org/wi... [15:51:49] ottomata: yt? [16:00:21] ping ottomata standdduppp [16:06:05] 10Analytics, 10Analytics-Kanban, 10ExternalGuidance, 10Product-Analytics: [Bug] `init` and `mtinfo` event counts drop drastically since June 17 2019 - https://phabricator.wikimedia.org/T227150 (10Nuria) a:05Milimetric→03Nuria [16:43:35] ottomata: I am going to remove temporarily the "filter_wiki_hostname" function from refine.pp so I can backfill all chelsey's data, sounds good? I will also do code patch but that way she can get going with analisys, let me know if this sounds good [16:46:45] 10Analytics, 10Analytics-Kanban, 10ExternalGuidance, 10Product-Analytics: [Bug] `init` and `mtinfo` event counts drop drastically since June 17 2019 - https://phabricator.wikimedia.org/T227150 (10Nuria) I am going to: 1. change puppet so we do not apply the 3rd party filter 2. re-refine all data since Jun... [17:02:24] * elukey off! [17:05:20] 10Analytics, 10Analytics-Kanban, 10Discovery, 10Operations, and 2 others: Make hadoop cluster able to push to swift - https://phabricator.wikimedia.org/T219544 (10Ottomata) Eric needs the analytics-search user to be able to access the swift auth file so his Oozie jobs can upload to swift. analytics-search... [17:07:50] 10Analytics, 10Analytics-Kanban, 10Discovery, 10Operations, and 2 others: Make hadoop cluster able to push to swift - https://phabricator.wikimedia.org/T219544 (10Nuria) @Ottomata , +1 to that idea [17:53:00] (03PS1) 10Ottomata: Use JsonParser to parse event data rather than YAMLParser [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/521552 (https://phabricator.wikimedia.org/T227484) [17:56:45] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Refine JsonSchemaLoader should use JsonParser instead of YAMLParser to load JSON data - https://phabricator.wikimedia.org/T227484 (10Ottomata) [18:05:14] (03CR) 10Alaa Sarhan: [C: 03+1] Use config for wdqs host name [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/520903 (https://phabricator.wikimedia.org/T218710) (owner: 10Ladsgroup) [18:05:36] (03Abandoned) 10Alaa Sarhan: Pass wbqsHost to service instance [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/521379 (owner: 10Alaa Sarhan) [18:24:35] (03PS1) 10Ottomata: Merge input JSONSchema with Hive schema before using it to read raw input JSON data [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/521563 (https://phabricator.wikimedia.org/T227088) [18:27:13] (03CR) 10Ottomata: "Not yet tested..." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/521563 (https://phabricator.wikimedia.org/T227088) (owner: 10Ottomata) [18:28:15] ottomata: then when this is merged i will go ahead and re-refine translation data , ok? https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/521541/ [18:28:32] (03CR) 10jerkins-bot: [V: 04-1] Merge input JSONSchema with Hive schema before using it to read raw input JSON data [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/521563 (https://phabricator.wikimedia.org/T227088) (owner: 10Ottomata) [18:31:54] (03PS2) 10Ottomata: Merge input JSONSchema with Hive schema before using it to read raw input JSON data [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/521563 (https://phabricator.wikimedia.org/T227088) [18:32:05] nuria: fyi merged and applied your puppet patch to remove 3rd party filtering [18:33:01] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Make JSONSchema aware Refine merge in existing Hive schema to read data - https://phabricator.wikimedia.org/T227088 (10Ottomata) [18:58:47] !log re-refining ExternalGuidance events for July 2019 [18:58:48] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:40:14] hi a-team, are the notebooks machines working correctly? I ssh in, but not connect to the notebook [19:41:14] dsaez: recently chelsey had as similar problem. somehow her jupyter was upgraded in her venv, but the juptyter server had not been restarted [19:41:26] which machine, both? [19:41:42] notebook1003 [19:42:11] now, after 5 mins, I'm connected to the login page, but stucked there again. [19:43:01] hm your env looks like the old versions as it should be [19:43:56] hm, i don't see your jupyter server running.. [19:44:14] dsaez: am watching logs can you try again? [19:44:22] ok [19:44:50] 'waiting for localhost ...' [19:45:25] hmm i see the same thing on 1003 [19:45:27] 1004 is fine tho... [19:49:19] dsaez: does 1004 work for you? [19:50:10] (03PS1) 10Nuria: Refactoring eventlogging-specific hostname check [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/521577 (https://phabricator.wikimedia.org/T227150) [19:50:39] hm it doesn't for me now [19:50:43] it looks like the ldap server isn't responding [19:51:15] let me see [19:52:07] ottomata, nop, same than 1003 [19:53:42] yeah... [19:53:43] something is wrong [19:55:28] it's capitalism, all problems comes from there. [20:07:57] haha [20:13:21] hahahaha [20:15:46] i think it isn't just notebook [20:17:26] hue also [20:20:06] 10Analytics, 10Operations, 10netops, 10LDAP: LDAP ldap-ro.eqiad.wikimedia.org not reachable from Analytics VLAN - https://phabricator.wikimedia.org/T227611 (10Ottomata) [20:25:14] dsaez: https://phabricator.wikimedia.org/T227611 [20:25:36] 10Analytics, 10Operations, 10netops, 10LDAP: LDAP ldap-ro.eqiad.wikimedia.org not reachable from Analytics VLAN - https://phabricator.wikimedia.org/T227611 (10Ottomata) p:05Triage→03High [20:31:00] 10Analytics, 10Analytics-Kanban, 10ExternalGuidance, 10Product-Analytics, 10Patch-For-Review: [Bug] `init` and `mtinfo` event counts drop drastically since June 17 2019 - https://phabricator.wikimedia.org/T227150 (10Nuria) Ok, data for july is there, onto data for june now: 2019-07-01 266256 2019-07-02... [20:35:26] (03CR) 10Ottomata: "You also had a comment about regex result caching in https://gerrit.wikimedia.org/r/c/analytics/refinery/source/+/511934/3/refinery-core/s" (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/521577 (https://phabricator.wikimedia.org/T227150) (owner: 10Nuria) [20:35:31] 10Analytics, 10Product-Analytics, 10Readers-Web-Backlog: Hash all pageTokens or temporary identifiers from the EL Sanitization white-list for Web - https://phabricator.wikimedia.org/T226850 (10kzimmerman) p:05Triage→03High [20:36:03] ottomata: withdrawing regex comment! [20:38:34] (03CR) 10Ottomata: Refactoring eventlogging-specific hostname check (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/521577 (https://phabricator.wikimedia.org/T227150) (owner: 10Nuria) [20:40:00] :) [20:49:06] (03CR) 10Ottomata: "Actually, one more thought. Since we are talking about allowing some non-wiki hostnames in T227150, maybe we should just make this a gene" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/521577 (https://phabricator.wikimedia.org/T227150) (owner: 10Nuria) [20:54:21] ottomata, thx, which team is managing the LDAP? [20:54:49] 10Analytics, 10Product-Analytics, 10Readers-Web-Backlog (Tracking): Hash all pageTokens or temporary identifiers from the EL Sanitization white-list for Web - https://phabricator.wikimedia.org/T226850 (10MNeisler) [20:56:11] dsaez: SRE, this specific issue will need some insight from NetOps people [20:56:31] (who are also part of SRE) [20:57:24] gotta [21:23:25] 10Analytics, 10Discovery-Analysis, 10Product-Analytics, 10Reading-analysis, 10Patch-For-Review: Productionize per-country daily & monthly active app user stats - https://phabricator.wikimedia.org/T186828 (10kzimmerman) a:05chelsyx→03None