[00:08:45] 10Analytics, 10Analytics-Kanban, 10Analytics-SWAP, 10Product-Analytics, 10Patch-For-Review: Publishing html files generated on notebook hosts - https://phabricator.wikimedia.org/T217619 (10chelsyx) Thank you so much @Ottomata ! Works like a charm! [00:22:21] 10Analytics, 10ExternalGuidance, 10Product-Analytics, 10MW-1.33-notes (1.33.0-wmf.18; 2019-02-19), 10Patch-For-Review: Measure the impact of externally-originated contributions - https://phabricator.wikimedia.org/T212414 (10chelsyx) **Update:** The url to the report is now changed to https://analytics.wi... [00:30:17] 10Analytics, 10Analytics-Kanban, 10Analytics-SWAP, 10Product-Analytics, 10Patch-For-Review: Publishing html files generated on notebook hosts - https://phabricator.wikimedia.org/T217619 (10Nuria) Awesome, please @chelsyx would you be so kind to document here: https://wikitech.wikimedia.org/wiki/SWAP#Shar... [00:43:44] 10Analytics, 10Analytics-Kanban, 10Analytics-SWAP, 10Product-Analytics, 10Patch-For-Review: Publishing html files generated on notebook hosts - https://phabricator.wikimedia.org/T217619 (10chelsyx) Hi @Nuria, @Tbayer already documented it. :) [01:03:43] 10Analytics, 10Community-Tech, 10SVG Translate Tool, 10Community-Tech-Sprint: Integrate Piwik with SVG Translate to keep track of metrics - https://phabricator.wikimedia.org/T215478 (10Samwilson) This has been included in [[ https://github.com/wikimedia/svgtranslate/releases/tag/0.4.0 | version 0.4.0 ]] an... [02:04:26] 10Analytics, 10Knowledge-Integrity, 10Research, 10Epic, 10Patch-For-Review: Citation Usage: run third round of data collection - https://phabricator.wikimedia.org/T213969 (10RyanSteinberg) The instrumentation code only reports extClick events on links explicitly coded with class `external`. It's simple t... [02:09:08] 10Analytics, 10Community-Tech, 10SVG Translate Tool, 10Community-Tech-Sprint: Integrate Piwik with SVG Translate to keep track of metrics - https://phabricator.wikimedia.org/T215478 (10Nuria) I can see couple visits, closing ticket. [02:09:17] 10Analytics, 10Community-Tech, 10SVG Translate Tool, 10Community-Tech-Sprint: Integrate Piwik with SVG Translate to keep track of metrics - https://phabricator.wikimedia.org/T215478 (10Nuria) 05Open→03Resolved [02:10:40] 10Analytics, 10Community-Tech, 10SVG Translate Tool, 10Community-Tech-Sprint: Integrate Piwik with SVG Translate to keep track of metrics - https://phabricator.wikimedia.org/T215478 (10Nuria) Ah, sorry, user for the piwik site is same than event metrics site on piwik, let us know if you cannot login and ac... [05:21:13] PROBLEM - Check the last execution of refinery-sqoop-mediawiki on an-coord1001 is CRITICAL: CRITICAL: Status of the systemd unit refinery-sqoop-mediawiki [05:50:38] (03PS3) 10Ottomata: Add SparkSchemaLoader capabilities to Refine and RefineTarget [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/494831 (https://phabricator.wikimedia.org/T215442) [06:28:59] morninggg [06:43:30] so we have 4 hadoop workers down in the production cluster [06:43:33] and some in the testing [06:43:39] I believe due to a networking change [06:47:50] ok need to wait for somebody to ack it and tell me if my fix is correct [06:48:01] in the meantime, the notebooks have disk full [06:48:09] dsaez: are you around by any chance? [06:48:55] ahhh wait the refinery cache is also an issue! [06:50:20] nevermind better now [06:50:30] just cleaned up old revs [06:50:46] I think that we might need to force scap to keep only one rev [06:52:57] (03PS1) 10Elukey: Keep only 1 rev to limit the amount of space occupied by scap [analytics/refinery/scap] - 10https://gerrit.wikimedia.org/r/494870 [06:54:24] i feel like you are up early, even considering CEST time zone :) [06:54:46] (03CR) 10Elukey: "Added also Tyler to know if this is a silly change (scap wise) or not :)" [analytics/refinery/scap] - 10https://gerrit.wikimedia.org/r/494870 (owner: 10Elukey) [06:55:30] hare: I usually wake up very early! And take a long break at midday :) [06:56:28] The long break at midday does sound very European :D [06:56:47] (Americans are greatly in need of Mediterranean work culture) [06:58:29] usually in the morning my girlfriend wakes up early and my cat doesn't allow me to sleep more than this hour :D [06:58:55] the long break is needed to avoid doing 12 hours per dat [06:58:57] *day [06:59:00] that is not healthy :D [07:26:22] 10Analytics, 10DBA, 10Operations, 10ops-eqiad, and 2 others: rack/setup/install labsdb1012.eqiad.wmnet - https://phabricator.wikimedia.org/T215231 (10Marostegui) @elukey the problem is that if we add it to the existing proxies, they'll be reachable by wikireplica users, as there is a round robin there, so... [07:44:10] Morning elukey, here :) [07:44:14] hare sorry [07:44:24] bonjour! [07:44:32] elukey: man, bad morning today - May I help with nodes issues? [07:44:57] I am trying to allow an-coord1001 for labsdb1012 atm, need to wait for somebody with network fu to give me the +1 [07:45:02] nothing more that we can do :( [07:45:35] k - thanks for handling all that (as usual) - <3 [07:46:37] <3 [07:47:37] joal: about the ferm rules - you were saying that you'd like also the stat nodes to reach labsdb1012? [07:48:06] elukey: if feasible, that'd be great -- It would allow any analytics-allowed person to sqoop from labs as needed [07:48:27] ack [07:49:26] elukey: about the sqoop alert, is there anything needed to prevent it ringing again? [07:49:57] elukey: I'm going to manually sqoop failed tables, but about the prod job, anything needed? [07:50:03] joal: I can disable it if you want [07:50:37] joal: about the ferm rules.. all the hadoop workers will need to pull data from labsdb1012 right? [07:50:46] Absolutely correct! [07:50:50] ouch [07:51:07] :S Actually I should have thought about that at first round - my bad elukey [07:51:41] no no it just came up in my mind as well [07:51:55] the main problem is that we don't have a list of hadoop workers in hiera [07:54:15] right [07:54:16] hm [08:04:02] but we have ANALYTICS_NETWORKS :) [08:04:07] that is probably the right compromise [08:34:44] joal: so one setting to set for labsdb1012 is when the query will be killed because considered too long [08:34:54] atm is 4h [08:35:05] sounds ok for us elukey [08:35:25] if 1 query take more than 4 hours, we should make it smaller :) [08:36:42] 10Analytics, 10Core Platform Team Backlog (Watching / External), 10Services (watching): Evaluate using TypeScript on node projects - https://phabricator.wikimedia.org/T206268 (10Physikerwelt) I have been playing with typescript for a while. For me, the biggest disadvantage is that one has to compile to run t... [08:37:13] I don't even know how this happened: on eswiki_p on labs, rev_comment field has been dropped [08:37:16] :S [08:46:20] (03CR) 10Joal: Update LogHelper for parameterization (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/493237 (owner: 10Joal) [08:46:31] (03PS3) 10Joal: Update LogHelper for parameterization [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/493237 [08:48:38] (03PS4) 10Joal: Update LogHelper for parameterization [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/493237 [08:50:59] (03PS2) 10Joal: Update DataFrameToHive dynamic partition mode [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/471693 (https://phabricator.wikimedia.org/T164020) [08:52:56] 10Analytics, 10Operations, 10decommission, 10ops-eqiad, 10User-Elukey: Decommission analytics100[1,2] - https://phabricator.wikimedia.org/T205507 (10elukey) Proposal for fix: ` elukey@asw2-d-eqiad# show | compare [edit interfaces interface-range vlan-analytics1-d-eqiad] member ge-9/0/5 { ... } +... [09:05:06] (03CR) 10Joal: Add JsonSchemaConverter to spark package (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/492756 (https://phabricator.wikimedia.org/T215442) (owner: 10Joal) [09:05:18] (03PS4) 10Joal: Add JsonSchemaConverter to spark package [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/492756 (https://phabricator.wikimedia.org/T215442) [09:15:20] hadoop workers back up [09:15:29] \o/n [09:24:47] going afk people, will be online later on [09:24:51] ping me on the phone if you need me [09:26:57] Hi joal :D [09:27:12] Morning addshore :) [09:27:16] sorry for yesterday addshore [09:27:28] no problem, i never remember when you day off is :D [09:28:16] So, you remember the query you did for me regarding the duplication within labels aliases and descriptions on wikidata [09:29:02] Now I essentially want the same but 1#(within labels only) 2#(within aliases only) 3#(within descriptions only) 4#(within labels and aliases) :D [09:30:03] addshore: do we have a task for this (can't recall) [09:30:19] We didn't last time, let me make one this time [09:30:25] and I'll copy when you said last time into it too [09:30:53] addshore: I wonder if I sent you the code for the request or not [09:30:58] https://phabricator.wikimedia.org/T217821 [09:31:03] joal: nope, not that I could find [09:32:26] you gave me the last value on 2018-12-06 19:10:16 [09:32:29] let me look in the channel logs [09:34:40] addshore: I have it (I bless my saving process, even if it makes a mess in my WIP folder :) [09:34:46] :D [09:36:09] woo, and now it is in the ticket! [09:36:09] D [09:36:10] addshore: copied the thing to the task [09:36:11] :D [09:36:53] addshore: do you take it from there, or shall I help? [09:37:10] I can certainly give it a go :D [09:37:23] addshore: I'm here if needed ;) [09:40:23] joal: i should be in a notbook right? the question is, which one? ;) [09:40:36] spark yarn regular? [09:40:55] wait, pyspart? *looks at the snippet* [09:41:05] addshore: you can either go in a notebook ,or use spark2-shell in CLI (I prefer the latter) - spark-yarn regular should do the job yes :) [09:54:00] also joal is there a newer parquet? or? [09:54:15] yes sir, there is [09:54:43] hdfs dfs -ls /user/joal/wmf/data/wmf/mediawiki/wikidata_parquet [09:54:52] last line --> /user/joal/wmf/data/wmf/mediawiki/wikidata_parquet/20190204 [09:54:58] :) [09:56:59] (03CR) 10Joal: "Some comments, but overall approach is nice" (035 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/494831 (https://phabricator.wikimedia.org/T215442) (owner: 10Ottomata) [09:57:28] elukey: I have a question about the patch for scap cache_revs [09:57:46] elukey: is this number taking into account currently worked rev? [09:58:07] elukey: if so, it means rollbacking could be problematic, no? [10:40:20] !log Manually fixed sqoop issues [10:40:22] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:45:34] mediawiki-history job has started - we're back in normal track [11:19:07] joal: i destroyed your nice script, https://phabricator.wikimedia.org/P8168, but got my data :P https://phabricator.wikimedia.org/T217821#5007815 [12:04:01] (03PS1) 10Joal: Remove comment from sqoop queries [analytics/refinery] - 10https://gerrit.wikimedia.org/r/494912 [12:11:32] Hi GoranSM - I have a question about WDCM-ETL [12:13:19] GoranSM: I have seen it running frequently in the past days - Is it a testing phase? [12:38:40] joal: exactly, we can do 2 maybe [12:38:43] as initial step [12:39:05] elukey: I feel safer this way, yes :) [12:39:21] ack will amend, can you add the comment? [12:39:42] sure [12:40:14] (03CR) 10Joal: Keep only 1 rev to limit the amount of space occupied by scap (031 comment) [analytics/refinery/scap] - 10https://gerrit.wikimedia.org/r/494870 (owner: 10Elukey) [12:40:19] <3 [12:52:27] (03PS1) 10Mforns: Add matchIf function to routes [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/494919 (https://phabricator.wikimedia.org/T187806) [12:57:10] 10Analytics, 10Analytics-Kanban, 10User-Marostegui: Migrate users to dbstore100[3-5] - https://phabricator.wikimedia.org/T215589 (10Addshore) [13:14:38] (03PS2) 10Elukey: Keep only 2 revs to limit the amount of space occupied by scap [analytics/refinery/scap] - 10https://gerrit.wikimedia.org/r/494870 [13:15:35] (03CR) 10Joal: [C: 03+1] "Thanks elukey :)" [analytics/refinery/scap] - 10https://gerrit.wikimedia.org/r/494870 (owner: 10Elukey) [13:17:07] (03CR) 10Thcipriani: Keep only 2 revs to limit the amount of space occupied by scap (031 comment) [analytics/refinery/scap] - 10https://gerrit.wikimedia.org/r/494870 (owner: 10Elukey) [13:17:56] joal: --^ [13:17:59] very interesting [13:19:59] indeed elukey - I can't think of a rollback that was not automated [13:30:58] 10Analytics, 10Knowledge-Integrity, 10Research, 10Epic, 10Patch-For-Review: Citation Usage: run third round of data collection - https://phabricator.wikimedia.org/T213969 (10bmansurov) I was not aware of the DOI case. Thanks for bringing it up. I think in that case it makes sense to use the URL only and... [13:51:37] 10Analytics, 10Core Platform Team Backlog (Watching / External), 10Services (watching): Evaluate using TypeScript on node projects - https://phabricator.wikimedia.org/T206268 (10Ottomata) Ha :) 5 seconds for tests sounds dreamy...come on over to the Java world...:p [13:52:44] !log manually killing mediawiki-history-denormalize-wf-2019-02 instead of letting it fail another 3 attemps [13:52:45] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:52:55] (03PS5) 10Joal: Refactor mediawiki-page-history computation [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/493390 (https://phabricator.wikimedia.org/T190434) [14:01:19] (03PS4) 10Mforns: Adapt codebase for new flake8 upgrade [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/494752 (https://phabricator.wikimedia.org/T215289) [14:07:37] (03PS5) 10Mforns: Adapt codebase for new flake8 upgrade [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/494752 (https://phabricator.wikimedia.org/T215289) [14:14:21] (03CR) 10Ottomata: [C: 03+1] "+1" [analytics/refinery/scap] - 10https://gerrit.wikimedia.org/r/494870 (owner: 10Elukey) [14:15:50] (03CR) 10Ottomata: [C: 03+2] Update LogHelper for parameterization [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/493237 (owner: 10Joal) [14:17:51] (03CR) 10Ottomata: [C: 03+1] Update DataFrameToHive dynamic partition mode [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/471693 (https://phabricator.wikimedia.org/T164020) (owner: 10Joal) [14:18:26] (03CR) 10Ottomata: [C: 03+2] Add JsonSchemaConverter to spark package [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/492756 (https://phabricator.wikimedia.org/T215442) (owner: 10Joal) [14:22:08] (03CR) 10Mforns: Adapt codebase for new flake8 upgrade (031 comment) [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/494752 (https://phabricator.wikimedia.org/T215289) (owner: 10Mforns) [14:22:13] (03CR) 10Mforns: [C: 03+2] Adapt codebase for new flake8 upgrade [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/494752 (https://phabricator.wikimedia.org/T215289) (owner: 10Mforns) [14:24:10] (03CR) 10Ottomata: Event(Logging) schema loader (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/492399 (https://phabricator.wikimedia.org/T215442) (owner: 10Ottomata) [14:36:46] (03CR) 10Ottomata: Add SparkSchemaLoader capabilities to Refine and RefineTarget (035 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/494831 (https://phabricator.wikimedia.org/T215442) (owner: 10Ottomata) [14:38:04] (03PS10) 10Ottomata: Event(Logging) schema loader [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/492399 (https://phabricator.wikimedia.org/T215442) [14:38:06] (03PS4) 10Ottomata: Add SparkSchemaLoader capabilities to Refine and RefineTarget [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/494831 (https://phabricator.wikimedia.org/T215442) [14:44:46] (03CR) 10jerkins-bot: [V: 04-1] Add SparkSchemaLoader capabilities to Refine and RefineTarget [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/494831 (https://phabricator.wikimedia.org/T215442) (owner: 10Ottomata) [14:46:21] (03PS5) 10Ottomata: Add SparkSchemaLoader capabilities to Refine and RefineTarget [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/494831 (https://phabricator.wikimedia.org/T215442) [14:48:56] elukey, yt? heya :] I'm trying to configure some RU jobs with the new auto_find_db_shard feature, and I found a job that uses a db called flowdb. Couldn't find it in the mediawiki-config dblists... I can configure it manually, pointing to the right host and port, but I don't know where it is, can you help me please? :] [14:50:16] mforns: hola! flowdb is on x1 [14:50:34] so you probably need to use use_x1=true or something similar [14:50:44] it should be in the x1.dblist though no [14:50:45] ? [14:50:46] !log Restart mediawiki-history after having corrected data [14:50:48] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:50:56] elukey, didn't find it there... [14:51:53] elukey, where is the x1 dblist? [14:56:48] gone for kids, see y'all at standup [15:06:19] (03PS2) 10Fdans: Create metrics matrix component [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/494546 [15:07:00] (03Abandoned) 10Fdans: Add getMainComponent to routes [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/494491 (https://phabricator.wikimedia.org/T187806) (owner: 10Mforns) [15:09:01] (03CR) 10jerkins-bot: [V: 04-1] Create metrics matrix component [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/494546 (owner: 10Fdans) [15:14:16] 10Analytics, 10Product-Analytics: Standardize datetimes/timestamps in the Data Lake - https://phabricator.wikimedia.org/T212529 (10Ottomata) > I'd be even more happy with standardizing on `YYYY-mm-dd HH:MM:SS` I'd like to hear from @JAllemandou (if he has an opinion!). But, I think we are a bit too far in fo... [15:22:59] mforns: so x1 contains the mw extensions, by default everything is in that shard (the dbs are smaller) [15:23:23] so if you know you want to use a mw extension db (like flow) you just need to set use_x1=true [15:33:04] (03CR) 10Bearloga: "New patch set coming up which addresses the feedback" (0310 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/493424 (https://phabricator.wikimedia.org/T209087) (owner: 10Bearloga) [15:33:56] (03PS3) 10Bearloga: Update whitelisting for Android-related schemas [analytics/refinery] - 10https://gerrit.wikimedia.org/r/493424 (https://phabricator.wikimedia.org/T209087) [15:42:47] elukey, yes use_x1 works, tested it [15:47:14] (03PS1) 10Mforns: Adapt config to use new auto_find_db_shard feature [analytics/limn-flow-data] - 10https://gerrit.wikimedia.org/r/494967 (https://phabricator.wikimedia.org/T215289) [15:47:46] mforns: nice! [15:51:41] hey elukey [15:51:55] were you looking for me? [15:54:30] (03PS2) 10Mforns: Adapt config to use new auto_find_db_shard feature [analytics/limn-flow-data] - 10https://gerrit.wikimedia.org/r/494967 (https://phabricator.wikimedia.org/T215289) [15:59:47] (03PS1) 10Mforns: Fix bug in auto_find_db_shard feature [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/494972 (https://phabricator.wikimedia.org/T215289) [16:01:41] (03PS1) 10Mforns: Adapt config to use new auto_find_db_shard feature [analytics/limn-edit-data] - 10https://gerrit.wikimedia.org/r/494974 (https://phabricator.wikimedia.org/T215289) [16:16:03] mforns: want me to merge those as you do them ^ ? [16:16:22] milimetric, mmmmm [16:16:54] milimetric, yes :] but please, merge this one first: https://gerrit.wikimedia.org/r/#/c/analytics/reportupdater/+/494972/ [16:17:05] I will add you as a reviewer [16:17:45] thank you! [16:18:08] (03CR) 10Milimetric: [C: 03+2] Fix bug in auto_find_db_shard feature [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/494972 (https://phabricator.wikimedia.org/T215289) (owner: 10Mforns) [16:19:41] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Adapt config to use new auto_find_db_shard feature [analytics/limn-flow-data] - 10https://gerrit.wikimedia.org/r/494967 (https://phabricator.wikimedia.org/T215289) (owner: 10Mforns) [16:20:18] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Adapt config to use new auto_find_db_shard feature [analytics/limn-edit-data] - 10https://gerrit.wikimedia.org/r/494974 (https://phabricator.wikimedia.org/T215289) (owner: 10Mforns) [16:31:58] milimetric, thanks :] [16:32:11] milimetric, do you recall any changes of permits related to the staging database? [16:32:33] mforns: not permits, just that it's on a separate shard [16:32:35] when reconfiguring the limn-language-data/cx job I get permit errors [16:32:37] the old one is read-only [16:32:46] yes, I believe the shard is ok [16:32:59] permit errors with the research user? [16:33:06] yes [16:33:15] what db is that accessing? [16:33:23] just staging? [16:33:34] staging [16:33:36] yes [16:34:13] maybe I should add a debug log that shows the host and port RU is connecting to for each query [16:34:25] 10Analytics: Sqoop: remove cuc_comment and join to comment table - https://phabricator.wikimedia.org/T217848 (10Milimetric) [16:35:08] (03CR) 10Milimetric: [V: 03+2 C: 03+2] "Merging with a couple of notes that we need to remember for the future." (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/494912 (owner: 10Joal) [16:35:48] mforns: well try to connect to it yourself first, it should be using the same credentials [16:42:17] milimetric, no! it's because it connects to log database as well... [16:42:30] connects to both... :[ [16:42:39] I don't see a solution to this.. [16:45:55] maybe migrating that job to hive [16:46:45] and adding a hive2druid job to enable turnilo and superset [17:01:29] (03CR) 10Elukey: [V: 03+2 C: 03+2] Keep only 2 revs to limit the amount of space occupied by scap [analytics/refinery/scap] - 10https://gerrit.wikimedia.org/r/494870 (owner: 10Elukey) [17:01:56] ping milimetric fdans [17:07:56] 10Analytics, 10Core Platform Team Backlog (Watching / External), 10Services (watching): Evaluate using TypeScript on node projects - https://phabricator.wikimedia.org/T206268 (10Milimetric) @Physikerwelt, that does suck, even if Java sucks more :) I thought maybe karma would be better, but they rejected the... [17:09:18] mforns: can't we just have two db configs, one for the log db and one for dynamic finding? [17:09:43] oh! no, we can't [17:09:46] it needs both at the same time... [17:09:51] milimetric, yes, but... the same query is, yea.. [17:41:06] 10Analytics: Sqoop: remove cuc_comment and join to comment table - https://phabricator.wikimedia.org/T217848 (10fdans) p:05Triage→03Normal [17:43:25] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Services (watching): EventGate (in k8s) takes a long time to load new schemas - https://phabricator.wikimedia.org/T217661 (10fdans) [17:43:41] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Services (watching): EventGate (in k8s) takes a long time to load new schemas - https://phabricator.wikimedia.org/T217661 (10fdans) p:05Triage→03High [17:47:17] 10Analytics: Make edit data lake data available as a snapshot on dump hosts - https://phabricator.wikimedia.org/T214043 (10fdans) [17:48:10] 10Analytics, 10Analytics-Kanban, 10Operations, 10vm-requests, and 2 others: Create an-tool1005 (Staging environment for Superset) - https://phabricator.wikimedia.org/T217738 (10fdans) [17:48:27] 10Analytics, 10Analytics-Kanban, 10Analytics-SWAP, 10Product-Analytics, 10Patch-For-Review: Publishing html files generated on notebook hosts - https://phabricator.wikimedia.org/T217619 (10fdans) 05Open→03Resolved [17:51:43] 10Analytics, 10Analytics-SWAP: heirloom-mailx fails trying to send out email from SWAP notebook - https://phabricator.wikimedia.org/T168103 (10mforns) p:05High→03Low [17:52:23] 10Analytics: AQS edits API should not allow queries without time bounds - https://phabricator.wikimedia.org/T189623 (10mforns) p:05High→03Normal [17:53:07] 10Analytics: Quantify volume of traffic on piwik with DNT header set - https://phabricator.wikimedia.org/T199928 (10mforns) a:03Nuria [17:55:20] 10Analytics: Fix download-project-namespace-map script to send alert if it fails - https://phabricator.wikimedia.org/T203824 (10mforns) p:05High→03Normal [17:57:56] 10Analytics, 10Analytics-EventLogging, 10Patch-For-Review: Resurrect eventlogging_EventError logging to in logstash - https://phabricator.wikimedia.org/T205437 (10mforns) 05Open→03Resolved This task is done. Resolving. [17:59:54] 10Analytics, 10Analytics-EventLogging, 10Patch-For-Review: Resurrect eventlogging_EventError logging to in logstash - https://phabricator.wikimedia.org/T205437 (10mforns) 05Resolved→03Open Re-opening, needs to check if it is deployed and restarted. [18:00:21] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: Resurrect eventlogging_EventError logging to in logstash - https://phabricator.wikimedia.org/T205437 (10mforns) [18:01:45] 10Analytics: Update datasets to have explicit timestamp for druid indexation facilitation - https://phabricator.wikimedia.org/T205617 (10mforns) p:05High→03Triage [18:02:11] 10Analytics, 10Analytics-Wikistats: Group pageview data per family in AQS so we can surface it in wikistats per-family pageview metrics - https://phabricator.wikimedia.org/T205730 (10mforns) p:05High→03Normal [18:02:43] 10Analytics: Percentage increase should be removed from"all" time range on wikistats UI - https://phabricator.wikimedia.org/T205809 (10mforns) p:05High→03Normal [18:03:09] 10Analytics: Update pageview_hourly to include timestamp for better druid indexation - https://phabricator.wikimedia.org/T208230 (10mforns) p:05High→03Triage [18:06:59] 10Analytics: Wikistats2 metric: top article creators - https://phabricator.wikimedia.org/T210423 (10mforns) p:05High→03Normal [18:08:26] 10Analytics, 10Wikimedia-Stream, 10Core Platform Team Backlog (Watching / External), 10Patch-For-Review, 10Services (watching): EventStreams process occasionally OOMs - https://phabricator.wikimedia.org/T210741 (10mforns) p:05High→03Triage [18:11:32] 10Analytics, 10Anti-Harassment, 10Product-Analytics: Add partial blocks to mediawiki history tables - https://phabricator.wikimedia.org/T211950 (10mforns) p:05High→03Normal [18:12:38] 10Analytics, 10Product-Analytics, 10Epic, 10User-Elukey: Provide feature parity between the wiki replicas and the Analytics Data Lake - https://phabricator.wikimedia.org/T212172 (10mforns) p:05High→03Normal [18:14:51] 10Analytics, 10Anti-Harassment, 10Product-Analytics: Add partial blocks to mediawiki history tables - https://phabricator.wikimedia.org/T211950 (10JAllemandou) Hi @nettrom_WMF, `ipblocks_restrictions` table is sqooped since this month on the cluster. However I think that `logging` table doesn't contain detai... [18:17:50] o/ ottomata [18:17:57] 10Analytics, 10Anti-Harassment, 10Product-Analytics: Add partial blocks to mediawiki history tables - https://phabricator.wikimedia.org/T211950 (10TBolliger) Partial blocks were only enabled on Italian Wikipedia in January of this year, so the only historical data would be from the past 2 months. [18:17:58] I'll be a bit late to the analytics hangtime. [18:18:08] Waiting for current meeting to end (10 more minutes) [18:18:48] Heya fdans - About wikistats2 deploy, shall I first deploy on beta or as the testing been done? [18:19:03] ottomata: I need another 1 min. [18:19:06] sorry for the delay [18:19:46] 10Analytics, 10Anti-Harassment, 10Product-Analytics: Add partial blocks to mediawiki history tables - https://phabricator.wikimedia.org/T211950 (10dbarratt) >>! In T211950#5009094, @JAllemandou wrote: > However I think that `logging` table doesn't contain detailed historical information on partial blocks. Th... [18:26:44] Hey, I have question regarding the X-Analytics header and it's values [18:27:22] do we need to urlencode the key&value when passing stuff to the XAnalytics header? [18:27:49] The example header right now: X-Analytics: mf-m=b%2Camc;ns=-1;special=MobileOptions;loggedIn=1 [18:28:26] the `,` character in mf-m is urlencoded, because of that PHP passes the `mf-m=b%2Camc` instead of `mf-m-b,amc`. is it ok? [18:28:47] sorry - instead of `mf-m=b,amc` [18:30:28] (03CR) 10Nuria: Event(Logging) schema loader (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/492399 (https://phabricator.wikimedia.org/T215442) (owner: 10Ottomata) [18:35:56] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: Resurrect eventlogging_EventError logging to in logstash - https://phabricator.wikimedia.org/T205437 (10phuedx) @mforns: I'm pretty sure this is deployed and (re)started as there are log lines appearing in Kibana for the followi... [18:39:36] I don't know, raynor, sorry, a-team any idea? it seems weird to me that a header is urlencoded, because it's not a URL, shouldn't be needed [18:39:57] I was just googling around mforns, I don't know off the top of my head either [18:40:30] raynor, where did you get the example header? [18:40:52] so the thing is that there is XAnalytics extension, but the MobileFrontend implements the XAnalytics header handling but itself o_O [18:40:58] I'm wondering what varnish does with it... I don't think it decodes it... [18:41:10] right [18:41:29] PROBLEM - eventbus grafana alert on icinga2001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. https://wikitech.wikimedia.org/wiki/EventBus [18:41:38] https://github.com/wikimedia/mediawiki-extensions-MobileFrontend/blob/master/includes/MobileFrontend.hooks.php#L111 [18:41:55] https://github.com/wikimedia/mediawiki-extensions-MobileFrontend/blob/master/includes/MobileContext.php#L1003 [18:42:06] it's already like, for pretty long time [18:42:27] previously it wasn't a problem, as the values were simple strings [a-z]+ [18:42:57] but now, we make a comma-separated string, so when we urlencode it, it will become `%2C` separated string [18:43:32] definitely, the MobileFrontend should use XAnalytics extension, instead of hardcoding everything [18:43:56] I'm looking at this line raynor: https://github.com/wikimedia/mediawiki-extensions-XAnalytics/blob/master/XAnalytics.class.php#L63 [18:44:03] trying to find out if that function urlencodes too [18:44:30] yep, it does: http://php.net/manual/en/function.http-build-query.php [18:44:31] RECOVERY - eventbus grafana alert on icinga2001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. https://wikitech.wikimedia.org/wiki/EventBus [18:45:01] so, raynor, I guess XAnalytics does it too, so it must mean everything is url-encoded as a query string [18:45:39] ok, yup, you're correct [18:45:53] ok, cool. thx [18:45:54] raynor: I don't think it hurts anything in our world if it's url-encoded, does it cause you problems? [18:46:32] also raynor, looks like http_build_query is better to use than the urlencode . = . urlencode pattern in MFE [18:47:12] later I'll use the XAnalytics extension [18:47:19] so most of the code from MF will go away [18:47:27] can I do that? do we use XAnalytics everywhere? [18:50:00] milimetric, ^ [18:50:54] raynor: I'm honestly not that familiar with how people use that in mediawiki extensions and/or core, but I think the idea is that they would all use that extension, right? [18:51:04] I think yes [18:56:06] I should become more familiar with this though [19:02:08] so, a bit of background - I wasn't familiar with that extension, Today I found it just by a surprise, I was reading the https://wikitech.wikimedia.org/wiki/X-Analytics#Code and there is a link to github [19:05:01] I was reviewing the https://phabricator.wikimedia.org/T212961 and just wanted to understand how the XAnalytics header work [19:05:11] 10Analytics, 10Anti-Harassment, 10Product-Analytics: Add partial blocks to mediawiki history tables - https://phabricator.wikimedia.org/T211950 (10nettrom_WMF) Hi @JAllemandou, >>! In T211950#5009094, @JAllemandou wrote: > However I think that `logging` table doesn't contain detailed historical information... [19:05:32] then I got confused by the urlencode bit, thats why I'm here :). milimetric, thanks for your time [19:05:41] I think everything is clear to me [19:06:45] hey, thanks for prompting me to look into it! [19:21:26] milimetric: about deploying wks2 --> I don't have access to test in beta [19:21:44] milimetric: site looks good locally, but since it;s my first time, I'm a bit afraid :) [19:25:24] sorry joal was away for a sec [19:25:31] np milimetric [19:25:36] 10Analytics, 10Anti-Harassment, 10Product-Analytics: Add partial blocks to mediawiki history tables - https://phabricator.wikimedia.org/T211950 (10JAllemandou) Hi folks - Thanks again for quick answers - My superbad I looked at the wrong. I confirm data is available. A first toward having it available in med... [19:26:18] milimetric: shall I bypass beta deploy, or not? [19:27:22] joal: giving you access now, just taking a while in horizon [19:27:29] awesome :) [19:28:03] ok joal you should have it now [19:28:32] I DO have it ;) Thanks milimetric [19:28:54] great [19:42:59] ok, WKS2 looks good on canary [19:45:11] milimetric: About versioning, given the git-log of things to be deployed, I'm assuming I'll go from 2.5.4 to 2.5.5 ? Or shall I go to 2.6.0? [19:49:04] also milimetric, running 'npm install' in the folder generate changes for package.json - I wonder if I should push those as part of my change or not [19:52:58] hm - I'm gonna get diner and then back [20:33:14] mforns: Heya - have you deployed wikistats2 before? [20:47:41] No wikistats2 expert around - I don't feel comfortable deploying as is - I'll wait tomorrow to get answers to my questions and will deploy next monday [20:53:00] Gone for tonight team - see you tomorrow [20:56:27] hey analytics folks! i'm having an issue with the formatting of the content translation JSON dumps -- who's the best person to talk to? [21:28:07] joal, I think I only did deploy wikistats2 as a copilot, but tomorrow we can do that together before standup if you want [22:01:24] (03PS1) 10Gehel: Publish both shaded and unshaded artifacts. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/495141 [22:02:28] (03CR) 10Gehel: "This is a proposal and should not be merged too fast. It changes the published artifacts, so it will probably break the deployment." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/495141 (owner: 10Gehel) [22:24:44] (03CR) 10Ottomata: "Haven't looked at changes but I like the idea in general! Thanks so much!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/495141 (owner: 10Gehel) [22:25:08] laters team!