[00:00:12] 10Analytics: Define battery of smoke tests to run by hand before realease - https://phabricator.wikimedia.org/T190837#4086808 (10Nuria) [01:26:38] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jan-Mar-2018): Investigate how to identify our code contributors to on-wiki code (gadgets, templates, modules) on a WMF site - https://phabricator.wikimedia.org/T190164#4086890 (10Quiddity) This is great, thanks! Questions: Above you write "use the si... [06:25:05] 10Analytics-Kanban, 10WikimediaUI Style Guide, 10Patch-For-Review: Setup & integrate analytics on design.wikimedia.org - https://phabricator.wikimedia.org/T188786#4087033 (10Prtksxna) @Volker_E Are we going to be using this for the landing page as well? [07:32:08] * xinbenlv Hi, we want to monitor a daily update of full list of wikipedia sites and number of articles. We can found non-realtime version here - https://en.wikipedia.org/wiki/List_of_Wikipedias (sites are added / removed by hand) and stats v1 sitemap of wikipedia: https://stats.wikimedia.org/EN/Sitemap.htm [07:32:31] but we couldn't find anything that is both daily and completed automated generated. [07:33:08] Thank you! [08:13:37] xinbenlv: hi! I checked https://wikimedia.org/api/rest_v1/ to see if there was anything specific for your use case, but didn't find a good answer. Later on during the day there will be more expert people to answer your question for sure :) [08:37:37] (03PS9) 10Fdans: Metrics carousel [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421001 (https://phabricator.wikimedia.org/T187440) [08:41:43] (03CR) 10Fdans: "Sounds good, abandoning change" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421041 (https://phabricator.wikimedia.org/T188953) (owner: 10Fdans) [08:42:18] (03Abandoned) 10Fdans: Adds "Load more rows..." UI button to table chart [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421041 (https://phabricator.wikimedia.org/T188953) (owner: 10Fdans) [08:44:42] (03PS1) 10Fdans: Adds "Load more rows..." UI button to table chart [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422375 (https://phabricator.wikimedia.org/T188953) [09:08:33] 10Analytics-Tech-community-metrics, 10Code-Health, 10Release-Engineering-Team (Kanban): Develop canonical/single record of origin, machine readable list of all repos deployed to WMF sites. - https://phabricator.wikimedia.org/T190891#4087229 (10Aklapper) [09:17:05] * elukey hates a bit zookeeper [09:27:36] (03CR) 10Fdans: [V: 032 C: 032] "Tested zoomed in, zoomed out, in both my external monitor and retina screen, in Mac Chrome and Firefox. LGTM, merging" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422102 (https://phabricator.wikimedia.org/T190782) (owner: 10Fdans) [09:37:44] so I haven't still found a good way to swap a zookeeper cluster safely [09:37:57] and the apache docs suggests that it is not safe to do it [09:38:15] up to version 3.5.something, that of course it is not even in Debian sid [09:38:27] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jan-Mar-2018): Investigate how to identify our code contributors to on-wiki code (gadgets, templates, modules) on a WMF site - https://phabricator.wikimedia.org/T190164#4087320 (10Aklapper) Hi @Quiddity, thank you for your interest. ~~Our new limited f... [09:38:29] there are currently two issues [09:39:39] 1) consistency between the zk cluster - namely prevent split brain scenario that can lead to data loss. Example: 3 nodes, two up to speed and one catching up. Two more nodes added, majority is 3: stale node + two new nodes (empty config) take the lead and decide to go on. == dataloss [09:40:44] 2) consistency in the zk clients - of course all the clients have a hardcoded list of zk servers to contact. So at some point, when changing the cluster, it must be taken into account that clients needs to contact $some node to do their ops [09:41:20] Since going from 3 to 6 nodes seems a big problem, I tried to scale it down to a simpler one [09:41:40] what happens if one of the conf100[123] nodes breaks badly for a hw failure? [09:41:45] how would we handle it? [09:42:11] Maybe doing one at the time is better (more cumbersome but..) [10:17:25] 10Analytics, 10Analytics-Wikistats: Improve scoping of CSS - https://phabricator.wikimedia.org/T190915#4087497 (10fdans) [10:18:01] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats 2.0: Page heading style varies - https://phabricator.wikimedia.org/T187412#4087514 (10fdans) I'm totally familiar with the issue you are describing, but it does seem like we corrected it at some point recently, since I'm now trying to replicat... [10:19:56] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jan-Mar-2018): Review entries in https://github.com/Bitergia/mediawiki-repositories/ to exclude/include and find out if still needed - https://phabricator.wikimedia.org/T187711#4087535 (10Aklapper) [10:19:58] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jan-Mar-2018): Review entries in https://github.com/Bitergia/mediawiki-repositories/ to exclude/include and find out if still needed - https://phabricator.wikimedia.org/T187711#3983039 (10Aklapper) 05Open>03Resolved Well. For Git in https://wikimed... [10:23:06] 10Analytics-Tech-community-metrics, 10Regression: Exclude upstream repositories in the default view on wikimedia.biterg.io (by setting up "Projects" once Bestiary is available?) - https://phabricator.wikimedia.org/T146135#4087550 (10Aklapper) Note that for proper Gerrit stats, applying a manual repo exclusion... [11:06:04] * elukey lunch! [12:12:04] (03PS3) 10Milimetric: Make geowiki job output monthly druid [analytics/refinery] - 10https://gerrit.wikimedia.org/r/422060 (https://phabricator.wikimedia.org/T190409) [12:12:25] (03Abandoned) 10Milimetric: [WIP] Add geowiki_monthly table [analytics/refinery] - 10https://gerrit.wikimedia.org/r/421616 (https://phabricator.wikimedia.org/T190409) (owner: 10Milimetric) [12:12:51] joal: https://gerrit.wikimedia.org/r/#/c/422060/ is ready to review/merge [12:13:11] I left a Pivot link as proof in the comments [12:13:23] fdans: ping me when/if you wanna work on the last patch together [12:13:54] milimetric: will review tomight after kids day :) [12:14:07] joal: no rush, just making sure you know [12:14:21] and I can self-merge and deploy too, no prob, the thing works [12:14:46] Awesome milimetric - Will probably merge and deploy tonight then :) [12:52:28] milimetric: hey, just back from lunch, want to pair? [12:53:12] fdans: start without me, I'm feeling a little queasy, gonna see if I can shake it off [12:53:28] ok [12:57:37] 10Analytics-Kanban, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), 10Patch-For-Review: Record and aggregate page previews - https://phabricator.wikimedia.org/T186728#4087956 (10mforns) Hey all, I started looking into aggregating event.virtualpageviews to wmf.virtualpageviews_hourly. What a... [13:21:22] hellloooo [13:21:23] joal: yt? [13:31:55] (03CR) 10Fdans: "Addressed the inline comments" (034 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421001 (https://phabricator.wikimedia.org/T187440) (owner: 10Fdans) [13:32:01] (03PS10) 10Fdans: Add carousel to browse through metrics in mobile view [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421001 (https://phabricator.wikimedia.org/T187440) [13:32:08] (03CR) 10jerkins-bot: [V: 04-1] Add carousel to browse through metrics in mobile view [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421001 (https://phabricator.wikimedia.org/T187440) (owner: 10Fdans) [13:35:49] (03PS11) 10Fdans: Add carousel to browse through metrics in mobile view [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421001 (https://phabricator.wikimedia.org/T187440) [13:37:19] 10Analytics-Kanban, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), 10Patch-For-Review: Record and aggregate page previews - https://phabricator.wikimedia.org/T186728#4088044 (10Jdlrobson) @mforns in theory we should never log events from redirects as we only ever show the preview of the can... [13:40:38] 10Analytics-Kanban, 10Patch-For-Review: Refresh SWAP notebook hardware - https://phabricator.wikimedia.org/T183145#4088049 (10Ottomata) Yeah, if this wasn't happening before, it is almost certainly due to the JupyterHub version upgrade. Should be fine since it goes through ssh. [13:52:03] (03PS12) 10Fdans: Add carousel to browse through metrics in mobile view [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421001 (https://phabricator.wikimedia.org/T187440) [13:53:13] Hey ottomata - Kids day today [13:53:22] ottomata: will be there from standup onwhile [13:54:22] hey fdans, ok, I'm still feeling shitty, let's DO THIS [13:54:23] lol [13:54:31] cave when you're ready [13:54:33] omw milimetric ! [13:58:57] * elukey afk for a bit! [14:00:20] 10Analytics, 10Analytics-Wikistats: Display of radio buttons in Wikistats 2 is somewhat confusing - https://phabricator.wikimedia.org/T183185#4088105 (10Amitjoki) a:03Amitjoki [14:10:12] (03PS1) 10Amitjoki: Bug T183185 Display of radio buttons in Wikistats 2 is somewhat confusing [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422407 [14:13:18] 10Analytics, 10Analytics-Wikistats, 10Patch-For-Review: Display of radio buttons in Wikistats 2 is somewhat confusing - https://phabricator.wikimedia.org/T183185#4088133 (10Amitjoki) Here's how it looks with the changes I've made - https://drive.google.com/file/d/1M7czDvN7Eqhf5DY6aIpDFMqyVX80oK3x/view. Als... [14:16:17] !log re-enabling replication of mediawiki job topics from main -> jumbo [14:16:19] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:29:00] fdans: so I'll chain my change after yours, and we can switch their orders if we decide later to do just responsive [14:29:39] sounds good [14:30:27] (03CR) 10Fdans: "> Some warnings on console " Vue warn]: Failed to resolve directive:" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421001 (https://phabricator.wikimedia.org/T187440) (owner: 10Fdans) [14:34:20] (03PS13) 10Fdans: Add carousel to browse through metrics in mobile view [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421001 (https://phabricator.wikimedia.org/T187440) [14:36:28] btw milimetric if you want to change the topic explorer first that'd be perfect, because right now the topic explorer is expanded by default and it shouldn't on mobile [14:37:41] fdans: why does it matter which I do first, I was just going to do them all and chain the change after yours, no? [14:38:25] ah, yes never mind [14:55:45] PROBLEM - Kafka main-eqiad consumer group lag for kafka-mirror-main-eqiad_to_jumbo-eqiad on kafkamon1001 is CRITICAL: CRITICAL: Group is in an error state. Worst Lag: eqiad.mediawiki.job.wikibase-addUsagesForPage/p0 - lag:480 offset:4578205881 [15:03:27] 10Analytics, 10Analytics-Cluster, 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: Move EventLogging analytics processes to Kafka jumbo-eqiad cluster - https://phabricator.wikimedia.org/T183297#4088266 (10Ottomata) [15:08:25] @elukey, thank you! [15:09:38] **elukey** I didn't found the our use case. We are mostly interested in the # of articles stats. But thanks! I will wait for other people if they have an idea. [15:11:05] 10Analytics, 10Services: Use --new.consumer for main codfw <-> eqiad Kafka MirrorMaker - https://phabricator.wikimedia.org/T190940#4088273 (10Ottomata) [15:11:17] 10Analytics, 10Services: Use --new.consumer for main codfw <-> eqiad Kafka MirrorMaker - https://phabricator.wikimedia.org/T190940#4088284 (10Ottomata) p:05Triage>03Normal [15:13:45] 10Analytics, 10Services: Use --new.consumer for main codfw <-> eqiad Kafka MirrorMaker - https://phabricator.wikimedia.org/T190940#4088288 (10Pchelolo) The changeprop/jobqueue only listens to local events in a DC, so having duplicate eqiad events in codfw and duplicate codfw events in eqiad would not be a prob... [15:14:00] 10Analytics, 10Services (watching): Use --new.consumer for main codfw <-> eqiad Kafka MirrorMaker - https://phabricator.wikimedia.org/T190940#4088290 (10Pchelolo) [15:14:43] 10Analytics, 10Services (watching): Use --new.consumer for main codfw <-> eqiad Kafka MirrorMaker - https://phabricator.wikimedia.org/T190940#4088291 (10Ottomata) Ah! right great. [15:18:51] xinbenlv: I'd suggest to send an email to analytics@ though! https://lists.wikimedia.org/mailman/listinfo/analytics [15:33:04] (03PS1) 10Milimetric: [WIP] Fixing some responsive details [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422422 [15:34:12] ottomata: ops sync :) [15:37:12] ooo [15:37:13] ya [15:37:14] one sec [15:48:51] 10Analytics, 10EventBus, 10Services (watching): Use --new.consumer for main codfw <-> eqiad Kafka MirrorMaker - https://phabricator.wikimedia.org/T190940#4088388 (10mobrovac) [15:53:22] PROBLEM - Kafka main-eqiad consumer group lag for kafka-mirror-main-eqiad_to_jumbo-eqiad on kafkamon1001 is CRITICAL: CRITICAL: Group is in an error state. Worst Lag: eqiad.mediawiki.page-delete/p0 - lag:1 offset:5581501 [15:57:22] 10Analytics-EventLogging, 10Analytics-Kanban: Hive EventLogging tables not updating since January 26 - https://phabricator.wikimedia.org/T186130#4088430 (10Nuria) [15:57:24] 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: Monitor and alert if no new data from JsonRefine jobs - https://phabricator.wikimedia.org/T186602#4088429 (10Nuria) 05Open>03Resolved [15:57:45] 10Analytics, 10Analytics-Cluster, 10Patch-For-Review: Port Kafka clients to new jumbo cluster - https://phabricator.wikimedia.org/T175461#4088433 (10Nuria) [15:57:47] 10Analytics, 10Analytics-Cluster, 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: Move EventLogging analytics processes to Kafka jumbo-eqiad cluster - https://phabricator.wikimedia.org/T183297#4088432 (10Nuria) 05Open>03Resolved [16:16:48] elukey: ok sort of good new about numDroppedMessages [16:16:48] https://github.com/axbaretto/kafka/blob/master/core/src/main/scala/kafka/tools/MirrorMaker.scala#L741-L747 [16:16:56] abort.on.send.failure defualts to true [16:17:01] so the MM process dies when this happens [16:17:42] (oo, that is not apache kafka link sorry) [16:18:13] https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/MirrorMaker.scala#L741-L747 [16:18:34] although, i'm not sure why we'd ever see it incremented if the process is supposed to die before incrementing... [16:18:49] oh, it just closes the producer [16:18:49] ok [16:18:50] yes [16:18:51] so [16:19:04] this is kind of ok, i might relax the alert after all [16:19:07] ottomata: you were looking for me earlier on - What's up ? [16:19:17] joal: wanted to talk to you about the sql based merge thing [16:19:26] hoodie? [16:19:33] there's a guy in NYC who plays bike polo who works for squarespace [16:19:49] he does very simliar work to us! [16:19:56] i talked to him about this, json -> hive stuff [16:20:10] he says they've written a spark job that does the same thing: read json, alter hive table, insert [16:20:23] i'm going to their office this evening to meet up with them and talk abou tit [16:20:33] was wondering if we should merge your sql merge thing first [16:20:34] Sounds great ottomata ! [16:20:35] so I can link to it [16:20:36] etc. [16:20:56] Works for me ottomat [16:20:58] Works for me ottomata [16:21:00] sorry [16:21:26] k, i think there were a few things still needed? there's some commented out code in there i think [16:21:46] hm - I wonder if the patch doesn't involve spark2 [16:23:27] Actually no - doesn't need spark2 [16:23:35] We can merge if you want [16:24:11] ottomata: What do think will happen out of this evening? Will we move to theirs or do we expect them to move to ours? [16:24:28] hah probably neither [16:24:33] i just want to see what they do, who's is better, etc. [16:24:37] ours is open source, theirs is not :p [16:24:49] Right [16:24:53] also want to talk about how they solve some of these problems in event data platform context [16:25:02] makes sense [16:25:03] they are sort of switching to avro and going through simliar pains [16:25:19] so, its mostly just a knowledge share, but i also think what we have is really cool now, so i want to show off a little :) [16:25:29] hehe :D Enjoy ;) [16:25:44] ottomata: I'd also like to spend some time testing hoodie (different topic) [16:26:12] I think it could really be super cool in helping us having almost up to date data for mediawiki replicas [16:26:27] hoodie...... [16:26:41] OH RIGHT [16:26:42] The idea is smart, implem seems a bit hacky, but he, if it does the job :) [16:26:42] right [16:26:43] this thing [16:27:26] elukey: https://gerrit.wikimedia.org/r/#/c/422430/1/modules/profile/manifests/kafka/mirror/alerts.pp [16:28:09] ottomata: Since I'm going to deploy tonight, do you want to merge the SQL-Generator stuff and I'll deploy refinery-source as well? [16:28:28] ottomata: ack thanks! [16:29:52] hm, joal were you going to deploy refinery source anyway? [16:30:07] I think it was not necessary [16:30:12] But can do nonetheless [16:30:27] joal: let's just merge and not deploy yet [16:30:32] works for me [16:30:35] I let you merge? [16:31:30] joal: i think there's some commented out code that still needs cleane dup [16:32:15] !log bouncing main -> jumbo mirror makers to increase heap size to 2G [16:32:16] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:32:21] ottomata: really? [16:32:31] PROBLEM - Kafka main-eqiad consumer group lag for kafka-mirror-main-eqiad_to_jumbo-eqiad on kafkamon1001 is CRITICAL: CRITICAL: Group is in an error state. Worst Lag: eqiad.mediawiki.job.htmlCacheUpdate/p0 - lag:176 offset:887521507 [16:32:52] Ah found it ottomata ! [16:33:03] hmmm [16:35:14] ottomata: I think the reason for which we would NOT want to use that version is non-explicit casting --> Stuff don't fail if types are incoherent, they try to be cast, and fail [16:35:35] (03PS3) 10Joal: Update Refine DataFrame converter [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/419217 [16:39:25] joal: ? [16:39:38] joal: meaning we won't know if they cast poorly? [16:39:50] correct ottomata [16:40:18] In the previous version, we didn't try to cast weird stuff into string for instance( sub-objects to string let's say) [16:40:35] that would probably fail, no? if you try to cast a struct? [16:40:37] or a map? [16:41:02] I think it converts it json style yes [16:41:59] FAILED: ClassCastException org.apache.hadoop.hive.serde2.typeinfo.MapTypeInfo cannot be cast to org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo [16:42:01] for a map [16:42:09] FAILED: ClassCastException org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo cannot be cast to org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo [16:42:10] for a struct [16:42:23] that's hive tho [16:42:23] hmm [16:42:29] switching to spark... [16:43:43] oof you are right joal [16:43:44] in spark [16:43:45] org.apache.spark.sql.catalyst.expressions.UnsafeMapData@1bbc89d7 [f701c0680,12800000004,9b,1,0,3cd,147,0,0,1,0,0,3ce,3cd,206,1300000000d,14000000006,0,14800000002,0,0,0,0,9b,113,cc,2fafbb73,29,0,0,0,0,0,0,0,0,0,77656976,772d302e31332e31,36322e666d,656c62617473,5355] [16:43:51] those are returned results [16:44:04] cast map as string gives [16:44:04] org.apache.spark.sql.catalyst.expressions.UnsafeMapData@1bbc89d7 [16:44:08] ottomata: I read the code doing that, I expected it to succeed indeed [16:44:09] and cast struct as string does [16:44:18] [f701c0680,12800000004,9b,1,0,3cd,147,0,0,1,0,0,3ce,3cd,206,130000000 ... [16:44:26] are these just toString methods or somethign? [16:44:36] interesting [16:44:54] ottomata: There are methods built to be able to ast [16:44:54] joal, i'm not sure which behavior is better :) [16:44:56] ottomata: There are methods built to be able to cast [16:45:03] its kinda nice not to fail [16:45:07] ottomata: Not sure either - That's why I tell :) [16:45:09] yeah [16:45:26] Hi ottomata ! Do you have a minute for some spark question? :) [16:45:49] joal: can you add a comment maybe with a git sha or github link with the commit that did the recursive type merging? explaining why we switched to sql casting instead? [16:45:56] chelsyx: sure! [16:46:43] ottomata: How can I use hive UDF when using spark to run the query? I've tried: [16:46:47] ottomata: I can do that - Just to be sure I'm not mistaking - We did that because of not willing to rewrite casters for every types we wanted to be able to convert [16:47:01] https://www.irccloud.com/pastebin/CxLqDccg/ [16:47:21] joal ya [16:47:50] k ottomata - Will add some comment in commit message [16:49:50] wow cool chelsyx didn't knowa bout this ArraySum thing [16:49:53] hm [16:51:58] chelsyx: i just did [16:52:04] pyspark2 --jars /srv/deployment/analytics/refinery/artifacts/refinery-hive.jar [16:52:09] then [16:52:10] spark.sql("CREATE TEMPORARY FUNCTION array_sum AS 'org.wikimedia.analytics.refinery.hive.ArraySumUDF'") [16:52:11] seemed happy [16:52:57] ottomata: Oh interesting! Thank you! [16:55:27] (03PS8) 10Joal: Update mediawiki-history spark job for performance [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/419516 (https://phabricator.wikimedia.org/T189449) [16:56:00] ottomata: another question: Is there a way to use spark from local machine? for example, I can create a ssh tunnel, and then connect to the mysql db and query from my laptop. Can I do this with spark? [16:56:03] milimetric: feeling ok for a ooie question? [16:56:10] joal: shoot [16:56:23] milimetric: No more daily table, right? [16:56:29] joal: I'm feeling weird, not bad! It's a funny story [16:56:42] ok milimetric :) [16:56:54] joal: no, I'm keeping the daily table because it's easier and lets people do fancier queries [16:57:04] but no daily druid [16:57:42] milimetric: Ah got it [16:59:11] so the sanitizing will have to drop old cu_changes partitions and old geowiki_daily partitions, but we'll do that once we decide what info to digest and keep, because we may want more stuff in the monthly table (at different activity levels. In that case we can reload, but I've thought that through and it seems fine [16:59:27] 10Analytics-Kanban, 10User-Elukey: Refresh zookeeper nodes in eqiad - https://phabricator.wikimedia.org/T182924#4088632 (10elukey) Noted one thing: when restarting the leader, an observer shuts down and tries to participate to leader election (like a participant) even if it should not: ``` 2018-03-28 16:52:51... [16:59:50] (as in, right now we're keeping 1-4, 5-99, 100+, but we may want 1-10, 11-20, etc. or whatever else was historically kept and we assumed we'd have with the daily table [16:59:51] ) [17:00:50] understood milimetric - Good for me - Merging ! [17:00:51] (03CR) 10Ottomata: [C: 032] Update Refine DataFrame converter [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/419217 (owner: 10Joal) [17:00:57] merged joal, thank you [17:01:10] ottomata: Ah - Wanted to add the comment and all [17:01:22] ottomata - didn;t do that fast enough :( [17:01:24] hmm, chelsyx i don't think so [17:01:35] spark isn't a server you can connect to, instead it is a process/job you launch directly [17:01:36] hm [17:01:54] (03CR) 10Joal: [V: 032 C: 032] "LGTM ! Merging for deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/422060 (https://phabricator.wikimedia.org/T190409) (owner: 10Milimetric) [17:02:44] ottomata: got you. maybe we have to have something like livy for that? https://livy.incubator.apache.org/ [17:02:48] ooo maybe chelsyxhttps://spark.apache.org/docs/latest/sql-programming-guide.html#distributed-sql-engine [17:02:48] hmmm [17:05:12] ok a-team - Everything merged on refinery for me - Giving you 5 minutes to ask me for more before starting to deploy :) [17:05:29] joal: make it rain skittles!!! [17:05:40] wait... what's the scope of what we're allowed to ask? [17:05:51] milimetric: You doesn't seem weirder than usual to me :) [17:06:02] * milimetric regroups to try again [17:06:12] :) [17:07:03] ottomata: About comments on the change between SQL-casts and manual-casts - Do you want me to add them in the code, and push a patch for that? [17:07:40] fdans: fyi the mapState takes as an optional parameter the "rest" of the computed definitions, so you can be a little more elegant there, but don't worry, when I go over this next quarter I'll do some spring cleaning [17:08:14] (03PS1) 10Amitjoki: T189195 Limit pan in Wikistats2 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422438 [17:08:42] sure joal! that would be great, sorry, i shouldn't have merged yet i guess [17:08:49] (03CR) 10Milimetric: [C: 031] "+2 it yourself whenever you're done looking at that scrolling issue (it's ok to not fix it in this patch)" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421001 (https://phabricator.wikimedia.org/T187440) (owner: 10Fdans) [17:08:55] np ottomata - Will do that [17:09:06] gonna go get some food [17:09:12] 10Analytics, 10Analytics-Wikistats, 10Patch-For-Review: Limit pan in Wikistats2 maps - https://phabricator.wikimedia.org/T189195#4088674 (10Amitjoki) a:03Amitjoki [17:14:24] (03PS2) 10Amitjoki: T189195 Limit pan in Wikistats2 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422438 [17:14:53] wow ok chelsyx so this is possible [17:15:35] spark comes with a 'jdbc thrift server' that allows you to connect to it via a jdbc client [17:15:44] the jdbc thrift server runs like any other spark job [17:15:48] so locally, or in yarn, or whateve [17:15:50] r [17:16:06] we aren't going to set up a regular one for general use, but you can run one yourself [17:16:08] 10Analytics-Kanban, 10User-Elukey: Refresh zookeeper nodes in eqiad - https://phabricator.wikimedia.org/T182924#4088681 (10elukey) As we did in a similar use case before (https://etherpad.wikimedia.org/p/Jessie-Zookeeper) the procedure should be to: 1) select one of conf100[456], add to it a new id in hiera,... [17:16:29] next month joseph and I hope to make spark2 the preferred default [17:16:36] along the way i'll probably update it to spark 2.3 [17:16:46] and when I do so, i'll include the thrift-server executable [17:17:04] then you'll be able to run it yourself (like any other spark2 execultable, pyspark2, etc.) [17:17:07] and connect to it [17:17:11] so it'll be something lke [17:17:21] spark2-thrift-server --master -yarn ... [17:17:22] etc. [17:17:31] then, make an ssh tunnel from your laptop [17:17:41] and you should be able to connect your GUI [17:17:49] to the thrift server process you started [17:18:27] ottomata: WOW! Sounds awesome!!! [17:18:45] ottomata: Thx!!! [17:19:25] chelsyx: actually you can do this now if you DL a spark2 distribution yourself, hmm, lemme get one on stat1005 real quick, let's try it [17:22:01] ottomata: not available in spark 2.2.? [17:22:06] it is it is! [17:22:13] but i didn't install the executlabe as part of the deb package [17:22:17] hey team, I'm back [17:22:26] Ah ottomata! Forgot about that [17:22:28] so you could figure out whatever java - jar invocation you need [17:22:29] :) [17:22:36] :) [17:23:41] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Google-Summer-of-Code (2018): Proposal: [Analytics] Improvements to Wikistats2 front-end - https://phabricator.wikimedia.org/T190949#4088715 (10Amitjoki) [17:24:57] 10Analytics-Kanban, 10Google-Summer-of-Code (2018): [Analytics] Improvements to Wikistats2 front-end - https://phabricator.wikimedia.org/T189210#4088742 (10Amitjoki) [17:24:58] chelsyx: i don't have a jdbc gui to try [17:25:00] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats Beta - https://phabricator.wikimedia.org/T186120#4088743 (10Amitjoki) [17:25:02] 10Analytics, 10Analytics-Wikistats, 10Patch-For-Review: Wikistat Beta: expand topic explorer by default - https://phabricator.wikimedia.org/T186335#3941423 (10Amitjoki) 05Open>03Resolved [17:25:38] i've started a thrift server on 1005 [17:25:45] can you try? [17:26:23] ssh -N stat1005.eqiad.wmnet -L 10000:stat1005.eqiad.wmnet:10000 [17:26:27] 10Analytics, 10Analytics-Wikistats, 10Patch-For-Review: Correct mixed case in Wikistats2 - https://phabricator.wikimedia.org/T190435#4088745 (10Amitjoki) 05Open>03Resolved [17:26:29] then connect your gui to [17:26:31] ottomata: I don't have a jdbc gui neither... For mysql, I connect from R using the Rmysql package [17:26:41] jdbc:hive2://localhost:10000 [17:26:44] ottomata: I will trying to figure out what I can do [17:26:45] oh [17:28:21] Ok a-team - Deploying [17:29:15] ok chelsyx i'm not going to keep this process running, i can show you how to run it at your leisure tho :) [17:29:27] !log Deploy refinery using scap [17:29:28] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:29:54] ottomata: Thanks so much! I have a meeting now. will ping you when i have a moment :) [17:30:11] k [17:30:54] PROBLEM - Kafka main-eqiad consumer group lag for kafka-mirror-main-eqiad_to_jumbo-eqiad on kafkamon1001 is CRITICAL: CRITICAL: Group is in an error state. Worst Lag: codfw.resource_change/p0 - lag:440 offset:5414934882 [17:35:43] * elukey off!! [17:37:11] Bye elukey [17:39:18] ottomata: Is there a way for us to solve the bizarre git state on stat1004? [17:39:24] hammer? [17:39:26] looking [17:39:36] ottomata: After dan's issue with git on stat1005, I don't want to use it anymore :) [17:40:11] joal: what is the weird status? [17:40:34] it looks ok? [17:40:56] when I cd to /srv/deployment/analytics/refinery [17:41:01] I have a bunch of errors [17:41:31] ottomata: https://gist.github.com/jobar/a77086e318c794812844b428f76652c5 [17:42:03] PROBLEM - Kafka main-eqiad consumer group lag for kafka-mirror-main-eqiad_to_jumbo-eqiad on kafkamon1001 is CRITICAL: CRITICAL: Group is in an error state. Worst Lag: codfw.resource_change/p0 - lag:3211 offset:5415126603 [17:42:06] grr ^ [17:42:09] v confused about this [17:42:13] um [17:42:49] joal: is that preventing deploy? i think it is ok now [17:42:56] i think maybe a git fat artifact was synging? [17:44:05] fixed ottomata - Thanks [17:44:17] ottomata: I was afraid of something not being ok in code [17:44:27] !log Deploying refinery onto hadoop [17:44:28] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:44:33] Thanks ottomata :) [17:49:51] (03CR) 10Nuria: "One question remaining, looks good on desktop." (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422375 (https://phabricator.wikimedia.org/T188953) (owner: 10Fdans) [17:51:47] 10Analytics, 10Analytics-Wikistats, 10Patch-For-Review: Display of radio buttons in Wikistats 2 is somewhat confusing - https://phabricator.wikimedia.org/T183185#4088808 (10Nuria) @Amitjoki feedback on commit message is a bit too verbose, do you think you could condense it a bit? Also, the "should not be do... [17:52:21] (03CR) 10Fdans: Adds "Load more rows..." UI button to table chart (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422375 (https://phabricator.wikimedia.org/T188953) (owner: 10Fdans) [17:53:03] PROBLEM - Kafka main-eqiad consumer group lag for kafka-mirror-main-eqiad_to_jumbo-eqiad on kafkamon1001 is CRITICAL: CRITICAL: Group is in an error state. Worst Lag: eqiad.mediawiki.revision-create/p0 - lag:215 offset:712229374 [17:53:43] 10Analytics, 10Analytics-Wikistats, 10Patch-For-Review: Display of radio buttons in Wikistats 2 is somewhat confusing - https://phabricator.wikimedia.org/T183185#4088814 (10Nuria) Please see: https://www.mediawiki.org/wiki/Gerrit/Commit_message_guidelines and note structure of commit message, we try to abide... [17:55:34] (03CR) 10Nuria: [C: 04-1] "Please see comments on ticket about commit message, it needs a bit of work" (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422407 (owner: 10Amitjoki) [17:58:15] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Correct documentation for the referer_class field in pageview_hourly - https://phabricator.wikimedia.org/T190579#4088827 (10JAllemandou) [18:00:23] !log Kill-Restart mediawiki-history-reduced-coord after deploy [18:00:25] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:00:54] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats Beta - https://phabricator.wikimedia.org/T186120#4088838 (10Nuria) @Amitjoki we do not close tasks until code is been deployed, we will do that once we do our next release. [18:01:18] 10Analytics-Kanban, 10Google-Summer-of-Code (2018): [Analytics] Improvements to Wikistats2 front-end - https://phabricator.wikimedia.org/T189210#4088845 (10Nuria) [18:01:20] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats Beta - https://phabricator.wikimedia.org/T186120#4088846 (10Nuria) [18:01:22] 10Analytics, 10Analytics-Wikistats, 10Patch-For-Review: Wikistat Beta: expand topic explorer by default - https://phabricator.wikimedia.org/T186335#4088844 (10Nuria) 05Resolved>03Open [18:02:26] !log Kill-Restart mobile_apps-session_metrics (bundle killed, coord started) [18:02:27] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:02:52] (03CR) 10Nuria: [C: 032] Adds "Load more rows..." UI button to table chart (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422375 (https://phabricator.wikimedia.org/T188953) (owner: 10Fdans) [18:02:58] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats Beta - https://phabricator.wikimedia.org/T186120#4088850 (10Amitjoki) @Nuria I'll remember it the next time :) Sorry for the inconvenience. [18:03:06] (03CR) 10Nuria: [V: 032 C: 032] Adds "Load more rows..." UI button to table chart [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422375 (https://phabricator.wikimedia.org/T188953) (owner: 10Fdans) [18:04:07] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Phasing away one of the mobile apps session metrics jobs. - https://phabricator.wikimedia.org/T190459#4074265 (10JAllemandou) [18:05:50] milimetric: I'm about to launch geowiki oozie stuff [18:05:54] elukey: we can talk about alarms sorry [18:05:58] milimetric: Do I need to create the hive tables? [18:06:10] joal, I'll do that [18:06:22] milimetric: I can do it - just wanted to know :) [18:06:43] joal: ok, then, everything in https://github.com/wikimedia/analytics-refinery/tree/master/hive/mediawiki/private [18:07:24] milimetric: Thanks !@ [18:07:32] joal: oh, I think maybe create_mediawiki_private_cu_changes_table.hql is already run [18:10:21] milimetric: Just checked that [18:10:23] It is indeed [18:10:28] k [18:10:45] Will run the ones for geowiki [18:11:04] PROBLEM - Kafka main-eqiad consumer group lag for kafka-mirror-main-eqiad_to_jumbo-eqiad on kafkamon1001 is CRITICAL: CRITICAL: Group is in an error state. Worst Lag: eqiad.mediawiki.job.globalUsageCachePurge/p0 - lag:1 offset:5623870 [18:12:44] joal: when are geowiki jobs run? [18:12:48] cc milimetric [18:13:07] nuria_: after data gets loaded [18:13:11] nuria_: normally after the sqooping, but these will have some catching up to do [18:13:28] milimetric: is scooping a cron that sets a success flag? [18:13:40] oh, joal, that makes me think of a good point, I should manually sqoop cu_changes for 2018-01 [18:13:42] milimetric: That was one question: Have we automoted the load for cu_changes (partitioned) ? [18:13:53] milimetric: Good catch [18:13:58] It's now or never ! [18:13:58] joal: yeah, it's running normally, but it started at 2018-02 [18:14:08] milimetric: ah so cron is set up [18:14:14] milimetric: for sqoop, yes - What about hive? [18:14:17] yes, it is, I set that up a while ago [18:14:33] milimetric: and cron creates success flag that triggers oozie job, right? [18:14:34] joal: right, there's another partitioning script, right? [18:14:54] nuria_: yeah, there's a miss here, there's another step I wrote in my checklist [18:15:17] to add _PARTITIONED flag, which triggers, right now there's only a _SUCCESS flag [18:15:55] milimetric: and what is the _PARTITIONED flag used for? [18:16:14] nuria_: sqoop creates data, but dsoesn't add partitions in hiuve [18:16:17] nuria_: I'm less familiar but I think it's after the repair table is run [18:16:34] Given data is partitioned by wiki, we don't recreate them manually, but ask hive to repair the ta [18:16:37] le [18:17:02] k, so I'll find that in puppet and submit a patch, but for this month I could do it manually too, so we don't get slowed down by these details [18:17:11] but they're all documented in my task: https://phabricator.wikimedia.org/T190409 [18:18:21] milimetric: ya, let's submit a patch so we do not forget, i still do not understand who "adds partitions in hive" forgive my thickness [18:18:42] we won't forget, it's an item in my checklist [18:18:56] but there must be a cron job in puppet that runs the partition tool that marcel wrote? [18:18:57] just a guess [18:19:05] milimetric: It's not puppet, it;s oozie [18:19:40] milimetric: It's oozie for other mediawiki tables [18:19:54] milimetric: oozie/mediawiki/history/load [18:20:19] joal: ah so after success flag from cron another oozie runs some partitioning code [18:20:28] Correct nuria_ [18:20:36] ah! I didn't know that joal [18:20:39] joal: ok, that makes more sense [18:20:45] nuria: So that Hive tables know about the partitions [18:21:12] ok, cool, so I need another patch and another deploy then :( sorry I didn't know [18:21:20] then, maybe I'll just do it manually now? [18:21:25] joal: i still do not understand fully though, cause if partitios are set up upon table creation say (wiki, yar, month) [18:21:45] joal: don't those need to be set up just once? [18:21:51] nuria_: when you write to the directories directly it doesn't update the partition metadata [18:22:16] nuria_: that only works if you do an "insert" statement into the table specifying the partitions as part of the insert [18:22:21] milimetric: ah i get it now! [18:22:38] nuria_: yeah, like if you look at this insert: https://gerrit.wikimedia.org/r/#/c/422060/3/oozie/mediawiki/geowiki/monthly/insert_geowiki_monthly_data.hql [18:22:54] that wouldn't need further partitioning, but if you sqoop into the directories then you need to repair the tables [18:22:57] bbiab [18:22:59] I just didn't know they set it up as an oozie job [18:26:12] milimetric: ok, got it now, thank you. [18:27:36] milimetric: i can help doing dashboards later today once data makes it into the monthly table [18:28:14] milimetric: in druid [18:28:55] nuria_: only if you want to do it, I have time to do that [18:29:04] To finish on the matter of partitions - The reason we use a repear and not an insert is because of the number of partitions [18:29:09] just didn't want to also merge/deploy more changes to oozie in too rushed a manner [18:29:26] oh yeah, it makes sense joal [18:29:58] milimetric: I'm super fine if for this month we manullay repair the tables and fleg the folders [18:30:07] this month = april [18:30:09] yep [18:30:13] great [18:30:37] milimetric: about data permissions - geo-wiki-daily is same as webrequest - What about geowiki-monthly? [18:31:54] joal: permissions? [18:32:15] access permission milimetric [18:32:21] jajaja [18:32:22] that sounds like something I don't know about, I thought hdfs accesses everything [18:32:36] milimetric: my question is for non-hdfs people :) [18:33:09] joal: I'm saying I don't know anything about how we do that, but all this data is available on stat1006, so statistics-users would have access to it [18:33:43] milimetric: I'll put it in he same group as webrequest for now [18:33:55] We'll relax if needed [18:34:33] joal: I'd appreciate any links to docs about that [18:34:43] joal: aham.. where are those groups per table? i though the directory was what dictated access [18:34:50] milimetric: tha means parent folder is owned/grouped by hdfs:analytics-privatedata-users [18:35:11] correct nuria_ - I'm changing directory rights [18:35:28] oooh, ok, I didn't know we had different permissions on different directories [18:35:32] cool, good to know [18:35:33] milimetric: with rights: drwxr-x--- [18:36:05] joal: let me know if you need a second pair of eyes on stuff like that [18:36:18] !log Making hdfs://analytics-hadoop/wmf/data/wmf/mediawiki_private accessible only by analytics-privatedata-users group (and hdfs obviously) [18:36:22] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:36:30] milimetric: If you want :) [18:36:46] milimetric: I'm gonna issue the change, you chek it's correct aftewr? [18:37:24] joal: oh I trust you, just if you wanted to do it in pairs [18:37:35] milimetric: We can do that ! [18:37:38] To the cave! [18:37:45] omw [18:38:13] 10Analytics-Kanban, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), 10Patch-For-Review: Record and aggregate page previews - https://phabricator.wikimedia.org/T186728#4088974 (10Jdlrobson) @mforns We use wgPageTitle for source title thus all titles will be canonical... ... with one one edge... [18:40:59] 10Analytics-Kanban, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), 10Patch-For-Review: Record and aggregate page previews - https://phabricator.wikimedia.org/T186728#4088988 (10mforns) @Jdlrobson That's why I mentioned that. I understand you want to equate pagePreviews with pageViews as muc... [18:56:22] Anyone in analytics work with lucas@wmde? https://phabricator.wikimedia.org/T190415 is an access request for analytics-privatedata-users access [18:56:39] we need someone who understands his work to review what he is requesting and sponsor the request [18:57:32] robh: i think you might need addshore , we can help after he takes a look at request and verifies it is pertinent. [18:57:53] robh: addshore works for DE office and he is very familiar with our systems [19:01:25] robh: i have comented on ticket, let's hold on for now [19:01:43] thank you! [19:02:01] i really wasnt sure who to ask, its much appreciated [19:02:14] PROBLEM - Kafka main-eqiad consumer group lag for kafka-mirror-main-eqiad_to_jumbo-eqiad on kafkamon1001 is CRITICAL: CRITICAL: Group is in an error state. Worst Lag: eqiad.mediawiki.job.refreshLinks/p0 - lag:331 offset:3034989021 [19:03:23] robh: ya, totally, please ask [19:06:45] yeah im not shy about demanding attention in random irc channels ;D [19:06:49] (03PS1) 10Milimetric: Fix bad import [analytics/refinery] - 10https://gerrit.wikimedia.org/r/422470 [19:07:07] (03CR) 10Milimetric: [V: 032 C: 032] Fix bad import [analytics/refinery] - 10https://gerrit.wikimedia.org/r/422470 (owner: 10Milimetric) [19:09:55] robh: GOOD, seems a core skill of your job [19:09:55] 10Analytics-Kanban, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), 10Patch-For-Review: Record and aggregate page previews - https://phabricator.wikimedia.org/T186728#4089067 (10Jdlrobson) I think we are good then. Here the page view to Barack Obama will also be tracked. Only the source url... [19:10:15] heh indeed [19:12:23] RECOVERY - Kafka main-eqiad consumer group lag for kafka-mirror-main-eqiad_to_jumbo-eqiad on kafkamon1001 is OK: OK: Group is in an error state. [19:13:51] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, 10Services (doing): CirrusSearchCheckerJob should have a title - https://phabricator.wikimedia.org/T190958#4089095 (10Pchelolo) p:05Triage>03High [19:16:26] 10Analytics, 10Analytics-Wikistats: Scroll should not bounce horizontally when swiping left/right - https://phabricator.wikimedia.org/T190959#4089108 (10fdans) [19:16:31] (03CR) 10Fdans: [C: 032] Add carousel to browse through metrics in mobile view [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421001 (https://phabricator.wikimedia.org/T187440) (owner: 10Fdans) [19:20:44] !log Start Geowiki jobs (monthly and druid) starting 2018-01 [19:20:46] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:21:53] !log synced refinery to hdfs (only python changes but just so we have latest) [19:21:54] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:29:04] ottomata: Do you have a moment to show me how to connect to 'jdbc thrift server'? :) [19:31:16] chelsyx: sure [19:32:02] ok chelsyx, in your homedir on stat1005 [19:32:07] i just put a copy of spark-2.3.0-bin-hadoop2.6 [19:32:10] cd into that dir [19:32:21] source spark-env.sh so that you have proper hadoop/spark conf env vars like: [19:32:25] source ./spark-env.sh [19:32:30] then, start the thriftserver like [19:33:02] ./sbin/start-thriftserver.sh --files /etc/hive/conf/hive-site.xml [19:33:11] that will start it in the background [19:33:14] to stop it later, do: [19:33:17] ./sbin/stop-thriftserver.sh --files /etc/hive/conf/hive-site.xml [19:33:30] if it works, and you want to launch it in yarn, so that you can query bigger stuff [19:33:34] Nice ottomata :) [19:33:37] you can pass it the same args you usually do to pyspark or spark-submit [19:33:38] Gone for tonight a-team [19:33:39] --master yarn, etc. [19:33:51] bye joal! [19:33:56] ottomata: I won't have the comments tonight - I'll try to think about it tomorrow [19:34:10] then, ssh tunnel to stat1005 like: [19:34:14] ssh -N stat1005.eqiad.wmnet -L 10000:analytics1001.eqiad.wmnet:10000 [19:34:19] (10000 is the default port it will use) [19:34:39] then you should be able to connect via jdbc to it with [19:35:37] !connect jdbc:hive2://localhost:10000 [19:35:38] sorry [19:35:39] just [19:35:42] jdbc:hive2://localhost:10000 [19:35:46] is the correct jdbc url [19:36:15] (03CR) 10Nuria: [V: 032 C: 032] Fix bad import [analytics/refinery] - 10https://gerrit.wikimedia.org/r/422470 (owner: 10Milimetric) [19:38:18] chelsyx: does that make sense? [19:38:47] ottomata: Yes! Thanks! I'm looking for a jdbc gui so that I can try :) [19:38:53] ottomata: I will let you know [19:39:06] ok cool [19:39:07] great! [19:39:22] !log bouncing main -> jumbo mirrormaker to apply increase in consumer num.streams [19:39:23] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:01:02] (03PS2) 10Milimetric: [WIP] Fixing some responsive details [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422422 [20:32:52] 10Analytics-Kanban, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), 10Patch-For-Review: Record and aggregate page previews - https://phabricator.wikimedia.org/T186728#4089294 (10mforns) OK, thanks! Another question: Pageview_hourly extracts its **project** field using the GetPageviewInfo UD... [20:45:31] 10Analytics-Kanban, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), 10Patch-For-Review: Record and aggregate page previews - https://phabricator.wikimedia.org/T186728#4089315 (10Ottomata) @mforns, @Jdlrobson and I discussed this. You should be able to use the `source_url` field on the Virtu... [20:45:45] 10Analytics-Kanban, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), 10Patch-For-Review: Record and aggregate page previews - https://phabricator.wikimedia.org/T186728#4089316 (10Jdlrobson) >We can add the target URL to the event (not even sure this is possible) Can we not use `source_url` fi... [20:46:53] 10Analytics-Kanban, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), 10Patch-For-Review: Record and aggregate page previews - https://phabricator.wikimedia.org/T186728#4089319 (10mforns) Oh... of course. Sorryyy [20:56:29] 10Analytics-Kanban, 10EventBus, 10Wikimedia-Stream, 10Services (watching), 10User-mobrovac: EventStreams - https://phabricator.wikimedia.org/T130651#4089331 (10Ottomata) [20:56:30] (03PS3) 10Milimetric: Fix responsive glitches [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422422 [20:56:34] 10Analytics, 10EventBus, 10Wikimedia-Stream, 10Services (watching), 10User-mobrovac: Bikeshed what events should be exposed in public EventStreams API - https://phabricator.wikimedia.org/T149736#4089329 (10Ottomata) 05Open>03Resolved Open too long, we mostly good here. :) [20:58:28] 10Analytics, 10Analytics-Cluster: Migrate EventStreams to Kafka Jumbo - https://phabricator.wikimedia.org/T189716#4089336 (10Ottomata) [20:58:30] 10Analytics, 10Analytics-Cluster, 10Patch-For-Review: Move EventStreams to new jumbo cluster. - https://phabricator.wikimedia.org/T185225#4089334 (10Ottomata) [20:59:26] 10Analytics: Put data needed for edits metrics through Event Bus into HDFS - https://phabricator.wikimedia.org/T131782#4089340 (10Ottomata) a:05Ottomata>03None I'm not totally sure what this task is, so I'm unassigning myself. [21:00:20] 10Analytics-EventLogging, 10Analytics-Kanban: Sunset MySQL data store for eventlogging - https://phabricator.wikimedia.org/T159170#4089346 (10Ottomata) [21:00:22] 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: Sanitize Hive EventLogging - https://phabricator.wikimedia.org/T181064#4089345 (10Ottomata) [21:00:25] 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: Lookout for duplicates in EL refine, implement pluggable transform method config in JSONRefine - https://phabricator.wikimedia.org/T185237#4089344 (10Ottomata) 05Open>03Resolved [21:05:02] (03PS4) 10Milimetric: Fix responsive glitches [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422422 [21:07:03] ok, nuria_, wikistats looks clean and good to go to me, but I don't know if there are other patches [21:07:06] check it out here: https://wikistats-canary.wmflabs.org/mobile [21:07:17] and we have to look at it on our phones, etc. [21:07:27] and the code review is this one: https://gerrit.wikimedia.org/r/#/c/422422/ [21:07:31] includes a list of what I did [21:07:57] cc fdans ^ [21:09:21] laters yall! [21:09:24] milimetric: will review tomorrow morning, hoping to deploy 2.2.0 in the afternoon :) [21:09:37] sweet, thanks fdans [21:15:21] milimetric: doing so now [21:15:44] milimetric: i will give it a pass with ipad and phones first [21:20:18] milimetric: gotta say on my new and shinny iphone looks delux, trying iphone4 now [21:23:13] milimetric: https://phabricator.wikimedia.org/T187440 [21:24:24] milimetric: swaping feels like you need to sap a lot to right or left to move [21:24:33] milimetric: the other two things [21:25:37] 10Analytics-Kanban: Allow switching metrics in a dashboard widget - Carrousel - https://phabricator.wikimedia.org/T187440#4089386 (10Nuria) {F16378091} When metric is not available, carrousel dots are faint, almost not visible. {F16378094} addblocker warning looks offplace in mobile view. [21:25:54] milimetric: added to ticket: https://phabricator.wikimedia.org/T187440 [21:57:49] 10Analytics-Kanban: Allow switching metrics in a dashboard widget - Carrousel - https://phabricator.wikimedia.org/T187440#4089488 (10Nuria) * responsive layout looks fine on both iphone4 dsafari and iphone8 chrome. On iphone8 looks real nice, specially detail page. * Swapping feels quite odd on both (understand... [22:23:34] (03PS5) 10Nuria: Fix responsive glitches [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422422 (https://phabricator.wikimedia.org/T187440) (owner: 10Milimetric) [22:35:19] (03PS3) 10Sturmkrahe: Darken subdued text and alter footer area for WCAG AA compliance [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/419958 (https://phabricator.wikimedia.org/T185533) [22:41:06] 10Analytics, 10Analytics-EventLogging, 10Performance-Team, 10Patch-For-Review: Make webperf eventlogging consumers use eventlogging on Kafka - https://phabricator.wikimedia.org/T110903#4089643 (10Krinkle) [22:43:33] 10Analytics, 10Analytics-Kanban: Add trash folder to hadoop - https://phabricator.wikimedia.org/T189051#4089665 (10Nuria) p:05Triage>03High a:03elukey [22:44:48] 10Analytics: Update UA parser - https://phabricator.wikimedia.org/T189230#4089682 (10Nuria) p:05Triage>03High [22:44:49] (03CR) 10Sturmkrahe: "> Why are we re-inventing the wheel here? The color palette" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/419958 (https://phabricator.wikimedia.org/T185533) (owner: 10Sturmkrahe) [22:45:22] 10Analytics: Update UA parser - https://phabricator.wikimedia.org/T189230#4035939 (10Nuria) a:03fdans [22:45:33] 10Analytics, 10Analytics-Kanban: Update UA parser - https://phabricator.wikimedia.org/T189230#4035939 (10Nuria)