[05:32:32] (03CR) 10Nuria: [C: 03+2] Tell map to render on chart type changes (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/526301 (https://phabricator.wikimedia.org/T226514) (owner: 10Milimetric) [05:33:27] (03Merged) 10jenkins-bot: Tell map to render on chart type changes [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/526301 (https://phabricator.wikimedia.org/T226514) (owner: 10Milimetric) [05:33:31] (03Merged) 10jenkins-bot: Remove unnecessary child reference [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/526302 (https://phabricator.wikimedia.org/T226514) (owner: 10Milimetric) [06:27:21] (03CR) 10Milimetric: Tell map to render on chart type changes (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/526301 (https://phabricator.wikimedia.org/T226514) (owner: 10Milimetric) [06:29:01] hello milimetric :) [06:29:19] hi :) [06:29:29] I'm about to go to bed, just had techcom and was catching up on any pings [06:29:55] it's nice to line up with you in the morning though :) [06:30:04] wish we were all in the same time zone [06:31:09] have a good night! [07:09:54] https://grafana.wikimedia.org/d/_rINM4DZk/varnishkafka-prometheus \o/ [07:10:18] brb [08:06:40] !log increase heap size on HDFS Namenodes (an-master100[12]) to 16G [08:06:42] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:28:58] PROBLEM - HDFS Namenode JVM GC CMS -olg gen- Heap usage percentage on an-master1001 is CRITICAL: 0.7094 ge 0.7 https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration%23HDFS_Namenode_Heap_settings https://grafana.wikimedia.org/dashboard/db/hadoop?var-hadoop_cluster=analytics-hadoop&panelId=87&fullscreen&orgId=1 [08:34:59] ah yes this is due to the alarm, it checks the past hour [08:35:05] will clear soon [08:36:52] The namenodes are now two 16G beasts :D [09:07:30] RECOVERY - HDFS Namenode JVM GC CMS -olg gen- Heap usage percentage on an-master1001 is OK: (C)0.7 ge (W)0.6 ge 0.56 https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration%23HDFS_Namenode_Heap_settings https://grafana.wikimedia.org/dashboard/db/hadoop?var-hadoop_cluster=analytics-hadoop&panelId=87&fullscreen&orgId=1 [10:37:28] * elukey lunch! [12:11:25] elukey: i notice puppet was complaining on stat1005 so i created https://gerrit.wikimedia.org/r/c/operations/puppet/+/526656. not sure if that is the right fix or not but if it is feel free to merge [12:48:41] jbond42: thanks a lot, it is indeed, I was waiting to fix the host before adding access to privatedata users (it is the first stat with buster) but forgot to ack :( [12:49:36] elukey: ahh ok, ill ignore it then :) [12:58:14] !log roll restart zookeeper on druid clusters with spicerack cookbook [12:58:16] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:06:45] morning! [13:07:41] o/ [13:10:36] hey all! [13:15:09] (03CR) 10Mforns: [V: 03+2 C: 03+2] "LGTM! Thanks for taking care of this!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525599 (https://phabricator.wikimedia.org/T226850) (owner: 10MNeisler) [13:27:59] o/ elukey what email does the kerberos pw reset email get sent by? [13:28:03] (i don't see it :) ) [13:29:20] it is the same that you got the last time [13:29:22] from kerberos1001 [13:29:31] have you ever received an email with the tmp pass? [13:29:55] Ah I do have it, it is from roo@ [13:29:55] because I recreated your account some days ago with the rest of the team [13:29:56] root@ [13:30:01] \o/ [13:30:02] which I filter and never read [13:30:11] I thought it was not working fiuuu [13:30:14] good :) [13:31:20] elukey: we should probably get them to send from a more official looking email addy eventually [13:32:16] makes sense yes [13:33:40] ok elukey running spark2-submit on an30 [13:33:51] 19/07/31 13:33:21 ERROR SparkContext: Error initializing SparkContext. [13:33:51] java.lang.IllegalArgumentException: requirement failed: A secret key must be specified via the spark.authenticate.secret config. [13:34:36] ah snap I think that the option is not needed [13:34:58] since we use yarn [13:35:12] ottomata: can you puppet disable and remove it from the default config? [13:35:13] oh, but what ab out local? [13:35:22] i'm running in local mode right now [13:35:37] IIUC the spark.authenticate.secret is needed when you want to use multiple nodes [13:35:40] but not via yarn [13:35:43] hm ok [13:35:50] https://spark.apache.org/docs/latest/security.html#authentication [13:35:56] For other resource managers, spark.authenticate.secret must be configured on each of the nodes. This secret will be shared by all the daemons and applications, so this deployment configuration is not as secure as the above, especially when considering multi-tenant clusters. In this configuration, a user with the secret can effectively impersonate any other user. [13:36:20] meanwhile [13:36:20] For Spark on YARN and local deployments, Spark will automatically handle generating and distributing the shared secret. Each application will use a unique shared secret. In the case of YARN, this feature relies on YARN RPC encryption being enabled for the distribution of secrets to be secure. [13:36:34] y reading too [13:36:43] oh 'and local' [13:36:51] so I should set spark.authenticate = False [13:36:52] ? [13:37:26] no I think that you can just remove spark.authenticate.secret from the default config [13:37:30] I am going to remove it from puppet [13:37:46] i don't see it there... [13:38:26] I was convinced that I added it, sigh [13:38:34] so maybe it is the default? [13:38:47] no it seems None [13:38:53] docs say default is false [13:39:16] ok ya you can repro simply with just spark2-shell [13:39:23] the shells tarts, but has failures for sure. [13:40:02] oh [13:40:03] elukey [13:40:04] https://issues.apache.org/jira/browse/SPARK-23476 [13:40:04] ottomata, I'm gathering info for today's deployment train, do you want the swift upload job to be deployed? [13:40:15] sigh [13:40:52] oh my i would love it ya, it'd also be nice if i could sneak in this kerberos auth change i'm making for refine for luca [13:40:54] hmm [13:41:00] but mforns they might not be ready. [13:41:07] actually the refine change is very simple [13:41:13] i just need to test it, i think i can in yarn mode, doing so now [13:41:14] ottomata, there's some comments on the swift one [13:41:17] yeah [13:41:22] let's not worry about thta one [13:41:26] that one will probably take some iterations [13:41:37] ok [13:42:28] let me know about the refine auth change, will add it to the train [13:43:03] (03PS1) 10Ottomata: DataFrameToHive: Set hive_server_url using full URL [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/526670 (https://phabricator.wikimedia.org/T228291) [13:43:07] mforns: it is ^ [13:43:11] but i neeed to test it before it is merged [13:43:22] elukey: to review ^^ too [13:43:27] ok! [13:43:43] I will confirm with you before deployment [13:45:29] (03CR) 10Elukey: [C: 03+1] DataFrameToHive: Set hive_server_url using full URL [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/526670 (https://phabricator.wikimedia.org/T228291) (owner: 10Ottomata) [13:45:33] (03CR) 10Mforns: [V: 03+2 C: 03+2] "LGTM! Thanks for taking care of this! Merging for this week's deployment train." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525280 (https://phabricator.wikimedia.org/T226852) (owner: 10Bearloga) [13:51:20] mforns: I can add the list of jobs to restart for the hive 2 actions to the etherpad if you want [13:51:45] elukey, that would be great :D [13:51:57] ack doing it [13:52:08] mforns: the cassandra bundle needs to be restarted right? [13:52:12] (Because I need to do the same) [13:52:37] elukey, yes, because of the unique_devices thing [13:52:56] ack, we can do it tomorrow [13:54:23] elukey, because it's 1st of month? [13:55:27] elukey: how do I use beeline? [13:55:28] i did [13:55:29] kinit [13:55:29] then [13:55:31] beeline [13:55:33] got [13:55:42] java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://analytics1030.eqiad.wmnet:10000: Peer indicated failure: Unsupported mechanism type PLAIN [13:56:40] got a simliar problem in spark i think [13:56:50] org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.transport.TTransportException [13:57:23] 19/07/31 13:57:03 ERROR log: Converting exception to MetaException [13:57:23] 19/07/31 13:57:03 WARN RetryingMetaStoreClient: MetaStoreClient lost connection. Attempting to reconnect. [13:57:23] MetaException(message:Got exception: org.apache.thrift.transport.TTransportException null) [13:57:41] (am runnign on an31) [13:57:54] ottomata: try to use an-tool1006 [13:58:09] it is the client [13:58:19] I never tested it yet on workers, will do [13:58:55] mforns: exactly [13:59:12] so we don't recompute too many things [14:00:12] elukey, ok [14:00:21] ok ya beeline works on an-tool1006 [14:00:50] milimetric, fdans, can this change be merged for deployment train? https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/522390/ [14:00:51] elukey: spark with hive not so much [14:01:01] spark2-shell --master yarn [14:01:06] val d = spark.sql("show databases") [14:01:14] I see a -1, but it seems prior to discussions in standup, so not sure! [14:01:16] Got exception: org.apache.thrift.transport.TTransportException null [14:01:25] mforns: no I think fran is still running those numbers [14:01:32] ok ok [14:01:35] thx [14:02:13] same result in spark2-shell local mode, with those extra authenticatae=false stuff from thta JIRA [14:02:36] I need to check [14:02:45] but spark2 still needs some love [14:02:51] this is why I asked for some help in testing :) [14:03:42] :) [14:03:45] let's give it <3 [14:03:49] elukey easiest way to repro: [14:03:55] spark2-thriftserver --conf "spark.authenticate=false" --conf "spark.shuffle.service.enabled=false" --conf "spark.dynamicAllocation.enabled=false" --conf "spark.network.crypto.enabled=false" --conf "spark.authenticate.enableSaslEncryption=false" [14:04:07] since thriftserver connects to hive directly [14:04:18] that will give you the same error in one command [14:04:21] in local mode [14:07:42] ottomata: do you need it now or can it wait tomorrow? [14:08:33] oh no it can wait! this is just for testing the refine change [14:08:43] it isn't needed for anything else [14:08:55] mforns: let's not hold up deployment for this then, i think we need a bunch more testing first [14:08:58] super, I'll try to fix it asap, it seems that also yarn mode doesn't work [14:09:07] right, both are the same [14:09:09] it seems [14:09:11] i get same error in yarn [14:09:21] just don't have to provide all those authenticate confs [14:09:28] ottomata, ok, no prob [14:09:35] I don't even get a yarn app id in running state though [14:09:36] strange [14:16:22] (03PS1) 10Mforns: Fix typo in "Improving examples arround how to start jobs" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/526681 [14:17:52] (03CR) 10Mforns: [V: 03+2 C: 03+2] "Self-merging to unbreak typo" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/526681 (owner: 10Mforns) [14:23:09] i got a flat tire! heading home but i need to get it fixed on the way bb in a bit [14:41:37] ottomata: sorry I didn't respond you yesterday while I was on the dentist, the thing to get out of the frozen ssh is pressing, in sequence (not at the same time): enter, tilde, period [14:50:45] ottomata: found the problem, basically an-tool1006 is missing an option in hive-site.xml that is needed [15:07:06] fdans, heya :] in the deployment train etherpad there is some mention of reportupdater migration, is that only to register what was done, or are there todos regarding RU? [15:07:14] https://etherpad.wikimedia.org/p/analytics-weekly-train [15:07:42] I mean TODOs for this deployment train [15:08:07] mforns: nothing to do on this train [15:08:19] I'm just giving people a couple more days before putting cleanup tags [15:08:34] mforns: thank you for asking :) [15:08:47] fdans, ok ok, should I move those lines to next train's docs in the etherpad? [15:10:10] mforns: I think everything up to future steps can be removed [15:10:33] oh, right will do [15:13:40] fdans, another thing :] I'm planning to deploy Wikistats today for: https://phabricator.wikimedia.org/T228937 and https://phabricator.wikimedia.org/T226514. Does this need any more testing, like in canary? [15:14:23] mforns: nope, you're good to go, thanks for doing that [15:14:31] k, thx! [15:37:21] !log starting deployment of analytics-refinery up to eb2d9b005b26f6dddab2b59f1ba591f1758ec99f [15:37:26] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:44:02] phew that took way longer than it should have! [15:44:56] elukey oh great! [15:48:22] (03PS1) 10Fdans: Fix pageviews agent type breakdown [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/526713 [15:48:28] mforns: ^ [15:49:27] (03CR) 10jerkins-bot: [V: 04-1] Fix pageviews agent type breakdown [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/526713 (owner: 10Fdans) [15:52:20] (03PS2) 10Fdans: Fix pageviews agent type breakdown [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/526713 [15:54:34] omg, has it been jerkins-bot instead of jenkins-bot all this time, I just noticed xD [16:04:18] !log finished deployment of analytics-refinery up to eb2d9b005b26f6dddab2b59f1ba591f1758ec99f [16:04:27] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:59:12] * elukey off! [17:11:22] (03CR) 10Nuria: "Sorry, thanks for catching this." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/526681 (owner: 10Mforns) [17:13:52] (03CR) 10Nuria: "Let's please test this jdbc connection string works before merging." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/526670 (https://phabricator.wikimedia.org/T228291) (owner: 10Ottomata) [17:16:36] 10Analytics, 10Operations, 10SRE-Access-Requests: Access to HUE for Mayakpwiki - https://phabricator.wikimedia.org/T229143 (10Nuria) @Mayakp.wiki hue has no ability to connect to druid (which is the data that powers both superset and turnilo), it can only connect to the hive datastore; To see sampling data... [17:37:38] nuria: one thing i remember about the task is that it was difficult to get folks to implement the 'solution needs to useable at scale.' [17:37:56] ottomata: ya, i added the "data does not fit in one machine" [17:37:59] yeah. [17:38:14] but ottomata any other ideas? [17:38:18] but, how would you do that anyway just as a programming task? without some big data system i.e. hadoop? [17:38:46] unless we maybe ask people to submit pseudo code and /or a description of their solution [17:58:12] (03CR) 10Mforns: [V: 03+2 C: 03+2] "LGTM!" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/526713 (owner: 10Fdans) [18:04:12] (03PS1) 10Mforns: Release 2.6.5 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/526740 [18:05:19] (03CR) 10Mforns: [V: 03+2 C: 03+2] "Merging for deployment" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/526740 (owner: 10Mforns) [18:05:59] !log deployed Wikistats2 version 2.6.5 [18:06:08] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:08:56] (03PS1) 10Ottomata: Deploy refinery to an-tool1006 in Hadoop test cluster [analytics/refinery/scap] - 10https://gerrit.wikimedia.org/r/526742 (https://phabricator.wikimedia.org/T228291) [18:09:46] (03CR) 10Ottomata: [V: 03+2 C: 03+2] Deploy refinery to an-tool1006 in Hadoop test cluster [analytics/refinery/scap] - 10https://gerrit.wikimedia.org/r/526742 (https://phabricator.wikimedia.org/T228291) (owner: 10Ottomata) [18:23:44] ottomata, can you merge this change? https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/519693/ it's pre-approved by luca, he just asked me to remove the --skip-trash argument from the call, which I did in the last patch set. [18:29:24] done mforns [18:30:27] ottomata, thanks! :D [18:36:21] a-team, I will go on an restart some oozie jobs, just FYI because of alarms [18:37:10] ok! [18:47:04] Analytics folks: can someone help us with https://phabricator.wikimedia.org/T216750#5370691 ? [18:48:47] leila: can me@usmanmuhd.com get +2 on this repo? that seems easiest [18:49:03] leila: did he filed for access? [18:50:20] nuria: re your first point, the volunteer is a GSoC volunteer. is it possible that I request that? [18:50:57] nuria: re the second point, I'm blanking. We did request for some privileges to remain but it seems this was not one of them. [18:50:57] leila: i am not sure , we can ping people in ticket [18:54:56] leila: pinged people on ticket [18:55:58] nuria: thanks! [19:00:47] mforns: let me know if you need any help with deploys or re-starts [19:01:06] thanks! [19:24:04] 10Analytics: placeholder - https://phabricator.wikimedia.org/T229464 (10Nuria) [19:29:33] 10Analytics, 10Analytics-Kanban: Load Netflow to Druid - https://phabricator.wikimedia.org/T225314 (10Nuria) The druid job is on but there is no data from My 24th onwards, @ayounsi , are you planning to enable the flow of this data in the near future? [19:29:42] 10Analytics, 10Analytics-Kanban: Load Netflow to Druid - https://phabricator.wikimedia.org/T225314 (10Nuria) [19:29:58] 10Analytics, 10Analytics-Kanban: Frequent/Long GC old gen collections for HDFS namenodes on an-master100[1,2] - https://phabricator.wikimedia.org/T228620 (10Nuria) 05Open→03Resolved [19:30:16] 10Analytics, 10Analytics-Kanban: Deal with truncated values in uniques - https://phabricator.wikimedia.org/T220098 (10Nuria) 05Open→03Resolved [19:30:23] 10Analytics, 10Analytics-Kanban: Deal with truncated values in uniques - https://phabricator.wikimedia.org/T220098 (10Nuria) Fix deployed now [19:31:54] 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, 10Product-Analytics: mediawiki_history missing page events - https://phabricator.wikimedia.org/T205594 (10Nuria) 05Open→03Resolved [19:31:56] 10Analytics, 10Analytics-Kanban: Mediawiki-history release - Snapshot 2019-06 - https://phabricator.wikimedia.org/T221825 (10Nuria) [19:31:58] 10Analytics-Kanban, 10Product-Analytics: Address data quality issues in the mediawiki_history dataset - https://phabricator.wikimedia.org/T204953 (10Nuria) [19:32:10] 10Analytics, 10Analytics-Kanban: Better error message for refine monitor so it takes into account that backfilling might be happening - https://phabricator.wikimedia.org/T228522 (10Nuria) 05Open→03Resolved [19:34:55] is there an easy to use eventgate container? I realized i cant test the new swift-upload.py on my local setup because i dont have an eventgate container in there [19:35:15] ottomata: ^ [19:35:29] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 2 others: Make it possible to use $ref in JSONSchemas - https://phabricator.wikimedia.org/T206824 (10Nuria) [19:35:37] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 2 others: Make it possible to use $ref in JSONSchemas - https://phabricator.wikimedia.org/T206824 (10Nuria) 05Open→03Resolved [19:35:39] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), 10Services (watching): Modern Event Platform: Schema Registry: Implementation - https://phabricator.wikimedia.org/T206789 (10Nuria) [19:36:42] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Core Platform Team Legacy (Watching / External), 10Services (watching): Factor out eventgate-wikimedia factory into its own gerrit repo and use it for deployment pipeline - https://phabricator.wikimedia.org/T226668 (10Nuria) 05Open→03Resolved [19:36:47] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Core Platform Team Legacy (Watching / External), 10Services (watching): Modern Event Platform: Stream Intake Service - https://phabricator.wikimedia.org/T201068 (10Nuria) [19:39:35] (03PS2) 10Ottomata: Refine: infer hiveServerUrl from config [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/526670 (https://phabricator.wikimedia.org/T228291) [19:39:42] ebernhardson: the contianer is in our registry [19:39:45] but also [19:39:50] mw-vagrant will set it up too [19:40:33] https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate#Development_in_Mediawiki_Vagrant [19:41:21] hmm i might need to check on ^ since I've modified the way we deploy eventgate to use a separate repo [19:42:14] ottomata: mw-vagrant doesn't have hadoop though :) I have some custom hodge-podge of docker-compose i use... [19:42:46] boots hadoop, kafka, mediawiki, master/slave mysql, zookeeper, mjolnir daemons, redis, etc. etc. [19:42:57] basically all the things cirrus talks to [19:43:11] (03PS3) 10Ottomata: Refine: infer hiveServerUrl from config [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/526670 (https://phabricator.wikimedia.org/T228291) [19:43:14] hmm [19:43:19] ottomata: ctrl-f for 'event' on https://docker-registry.wikimedia.org/v2/_catalog doesn't show any images/ [19:43:45] https://tools.wmflabs.org/dockerregistry/wikimedia/eventgate-wikimedia/tags/ [19:44:17] huh, wonder why the registry catalog doesn't have it [19:44:32] also [19:44:32] curl https://docker-registry.wikimedia.org/v2/wikimedia/eventgate-wikimedia/tags/list [19:44:38] dunno whata the _catalog is [19:45:16] its a semi-useless api based on spec :P Some registries may opt to provide a full catalog output, limit it based on the user’s access level or omit upstream results, if providing mirroring functionality. Subsequently, the presence of a repository in the catalog listing only means that the registry may provide access to the repository at the time of the request [19:45:21] (thats documentation quote) [19:45:29] ha ok [19:45:45] ebernhardson: i've never used that container outside of helm/k8s [19:46:10] dunno how the configs get passed to the container [19:46:26] i can probably figure out something, thanks! [19:46:29] ok! [19:47:25] (03PS4) 10Ottomata: Refine: infer hiveServerUrl from config [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/526670 (https://phabricator.wikimedia.org/T228291) [20:00:09] (03PS5) 10Ottomata: Refine: infer hiveServerUrl from config [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/526670 (https://phabricator.wikimedia.org/T228291) [20:01:56] (03CR) 10Ottomata: "Ok! Tested in both Hadoop clusters, with and without kerberos. This works and is way nicer; we can remove that extra annoying parameter " [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/526670 (https://phabricator.wikimedia.org/T228291) (owner: 10Ottomata) [20:24:50] (03PS6) 10Ottomata: swift-upload.py to handle upload and event emitting [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525435 (https://phabricator.wikimedia.org/T227896) [20:27:51] (03PS7) 10Ottomata: swift-upload.py to handle upload and event emitting [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525435 (https://phabricator.wikimedia.org/T227896) [20:32:59] (03PS8) 10Ottomata: swift-upload.py to handle upload and event emitting [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525435 (https://phabricator.wikimedia.org/T227896) [20:33:32] (03CR) 10Elukey: "Nice!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/526670 (https://phabricator.wikimedia.org/T228291) (owner: 10Ottomata) [20:34:55] (03CR) 10Ottomata: swift-upload.py to handle upload and event emitting (034 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525435 (https://phabricator.wikimedia.org/T227896) (owner: 10Ottomata) [20:59:13] (03CR) 10EBernhardson: swift-upload.py to handle upload and event emitting (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525435 (https://phabricator.wikimedia.org/T227896) (owner: 10Ottomata) [20:59:57] sorry :( minor thing i noticed while testing.. [21:00:07] (easy to fix, and i can otherwise workaround ofC) [21:29:17] mforns_brb: ok, i have looked at logs -again- (for cassandra inserts) and i do not see anything wrong there, also looked at reducer code that sends data to cassandra and that one works per line so i could not find any issues. [21:29:45] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 3 others: Options for implementing JobQueue statistics methods - https://phabricator.wikimedia.org/T175957 (10Pchelolo) 05Open→03Declined There has not been any complains about kafka queue not supporting these statistics, so I don't think... [21:29:49] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 5 others: [EPIC] Develop a JobQueue backend based on EventBus - https://phabricator.wikimedia.org/T157088 (10Pchelolo) [21:31:57] 10Analytics, 10ChangeProp, 10Core Platform Team, 10MediaWiki-JobQueue, and 2 others: Consider the possibility of separating ChangeProp and JobQueue on Kafka level - https://phabricator.wikimedia.org/T199431 (10Pchelolo) [21:32:16] (03PS9) 10Ottomata: swift-upload.py to handle upload and event emitting [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525435 (https://phabricator.wikimedia.org/T227896) [21:32:36] (03CR) 10Ottomata: swift-upload.py to handle upload and event emitting (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525435 (https://phabricator.wikimedia.org/T227896) (owner: 10Ottomata) [21:34:23] 10Analytics, 10ChangeProp, 10Operations, 10Core Platform Team Legacy (Designing), and 2 others: Separate dev Change-Prop from production Kafka cluster - https://phabricator.wikimedia.org/T199427 (10Pchelolo) 05Open→03Declined We don't really have/use change-prop in dev cluster anymore and I don't think... [21:39:08] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 3 others: Support dynamic rates for ChangeProp - https://phabricator.wikimedia.org/T188667 (10Pchelolo) 05Open→03Declined This is definitely overcomplication of the system. [21:39:12] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 5 others: [EPIC] Develop a JobQueue backend based on EventBus - https://phabricator.wikimedia.org/T157088 (10Pchelolo) [22:13:22] 10Analytics, 10Analytics-Kanban: Load Netflow to Druid - https://phabricator.wikimedia.org/T225314 (10ayounsi) Thanks ! I enabled it and added more dimensions, please let us know if there is any issue. `name=kafkacat output {"event_type": "purge", "tag2": 2, "as_src": 64600, "as_dst": 28271, "as_path": "2914_... [22:28:40] (03CR) 10EBernhardson: swift-upload.py to handle upload and event emitting (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525435 (https://phabricator.wikimedia.org/T227896) (owner: 10Ottomata) [22:44:46] 10Analytics, 10RESTBase, 10Core Platform Team Legacy (Later): REST API entry point web request statistics at the Varnish level - https://phabricator.wikimedia.org/T122245 (10Pchelolo) 05Open→03Declined In rare cases when we need it we can run a query on hadoop. [22:44:48] 10Analytics: Prototype counting of requests with real time (streaming data) - https://phabricator.wikimedia.org/T159264 (10Pchelolo) [22:44:55] 10Analytics, 10Core Platform Team, 10Developer-Advocacy, 10MediaWiki-API, and 4 others: Metrics about the use of the Wikimedia web APIs - https://phabricator.wikimedia.org/T102079 (10Pchelolo) [22:53:19] (03CR) 10Ottomata: swift-upload.py to handle upload and event emitting (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525435 (https://phabricator.wikimedia.org/T227896) (owner: 10Ottomata) [23:41:20] 10Analytics: Prototype counting of requests with real time (streaming data) - https://phabricator.wikimedia.org/T159264 (10Nuria) >Currently we search through the entire webrequest or webrequest_raw dataset for something that matches condition X The premise of this ticket is a bit old, rather than look through... [23:51:16] 10Analytics, 10Operations, 10SRE-Access-Requests: Access to HUE for Mayakpwiki - https://phabricator.wikimedia.org/T229143 (10Nuria) @Mayakp.wiki please give a try to jupyter and let me see on my end what is needed for access