[00:56:30] PROBLEM - Check the last execution of monitor_refine_eventlogging_analytics_failure_flags on an-launcher1001 is CRITICAL: CRITICAL: Status of the systemd unit monitor_refine_eventlogging_analytics_failure_flags https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [04:55:49] 10Analytics, 10Fundraising-Backlog: Dashboard for CentralNotice impression rates using Druid, centralnotice_analytics and CN events - https://phabricator.wikimedia.org/T254792 (10AndyRussG) [06:05:30] 10Analytics, 10Analytics-Kanban, 10Operations, 10observability, 10Patch-For-Review: systemd::syslog conf should use :programname equals instead of startswith - https://phabricator.wikimedia.org/T251606 (10elukey) @Ottomata I realized today that the issue pointed out by Marcel during standup (namely logs... [06:13:36] RECOVERY - Check the last execution of monitor_refine_eventlogging_analytics_failure_flags on an-launcher1001 is OK: OK: Status of the systemd unit monitor_refine_eventlogging_analytics_failure_flags https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [06:32:39] 10Analytics: Check home leftovers of lexnasser - https://phabricator.wikimedia.org/T252363 (10elukey) Copied the content of Lex's home dir on notebook1003 to stat1007: ` elukey@stat1007:/srv/home/lexnasser/notebook1003$ ls Geoeditors.ipynb hs_err_pid3889.log Search_Engine_Testing.ipynb source ` [06:33:12] 10Analytics: Check home leftovers of lexnasser - https://phabricator.wikimedia.org/T252363 (10elukey) 05Open→03Resolved p:05Triage→03Medium [06:34:37] 10Analytics, 10Documentation: Update Mediawiki Table manuals on wiki (Manual:Logging_table et al) - https://phabricator.wikimedia.org/T179407 (10Aklapper) [06:35:00] 10Analytics: Resetting Kerberos access for sguebo - https://phabricator.wikimedia.org/T254035 (10elukey) 05Open→03Resolved ` elukey@krb1001:~$ sudo manage_principals.py delete sguebo elukey@krb1001:~$ sudo manage_principals.py create sguebo --email_address=sguebo@wikimedia.org Principal successfully created.... [07:07:42] 10Analytics, 10Better Use Of Data, 10Event-Platform: Produce an instrumentation event stream using new EPC and EventGate from client side browsers - https://phabricator.wikimedia.org/T241241 (10elukey) >>! In T241241#6040663, @Ottomata wrote: > In meeting today we made a decision. To be clear, here's what t... [08:23:36] 10Analytics-Data-Quality, 10QuickSurveys, 10Research, 10WMDE-Technical-Wishes-Team, and 3 others: Remove Do Not Track support for QuickSurveys - https://phabricator.wikimedia.org/T254224 (10awight) [08:24:09] soooo Archiva automatically deletes old snapshots [08:24:13] not released artifacts sigh [08:25:04] I removed 10g of usage simply deleting old refinery-job releases [08:25:05] https://grafana.wikimedia.org/d/000000377/host-overview?panelId=28&fullscreen&orgId=1&refresh=5m&var-server=archiva1001&var-datasource=eqiad%20prometheus%2Fops&var-cluster=misc&from=now-1h&to=now [08:25:21] (manually via ui) [08:28:49] 10Analytics-Data-Quality, 10QuickSurveys, 10Research, 10WMDE-Technical-Wishes-Team, and 3 others: Remove Do Not Track support for QuickSurveys - https://phabricator.wikimedia.org/T254224 (10awight) [08:38:17] ok got down to 55G, looks good now [08:41:23] 10Analytics, 10Analytics-Kanban: Purge old files on Archiva to free some space - https://phabricator.wikimedia.org/T254849 (10elukey) 05Open→03Resolved a:03elukey While reading the documentation I discovered that snapshot artifacts are the ones easily deletable via settings, but released ones seem to be... [08:41:24] 10Analytics, 10Analytics-Kanban: Move Archiva to Debian Buster - https://phabricator.wikimedia.org/T252767 (10elukey) [08:41:35] 10Analytics, 10Analytics-Kanban: Purge old files on Archiva to free some space - https://phabricator.wikimedia.org/T254849 (10elukey) [09:07:17] 10Analytics-Data-Quality, 10QuickSurveys, 10Research, 10WMDE-Technical-Wishes-Team, and 3 others: Remove Do Not Track support for QuickSurveys - https://phabricator.wikimedia.org/T254224 (10WMDE-Fisch) Not as a blocker but I just wanted to mention that we currently "use" DNT to avoid that #quicksurveys "me... [09:43:33] 10Analytics, 10Analytics-Kanban, 10Operations, 10vm-requests: Create archiva1002 as replacement of archiva1001 - https://phabricator.wikimedia.org/T254890 (10elukey) ` elukey@cumin1001:~$ sudo cookbook sre.ganeti.makevm eqiad_A archiva1002.wikimedia.org --vcpus 4 --memory 4 --disk 100 --network public STAR... [10:15:16] 10Analytics-Data-Quality, 10QuickSurveys, 10Research, 10WMDE-Technical-Wishes-Team, and 3 others: Remove Do Not Track support for QuickSurveys - https://phabricator.wikimedia.org/T254224 (10ovasileva) @Lena_WMDE - looks like this is all done. Feel free to sign this off, or let us know if you would like u... [10:19:43] 10Analytics, 10Operations, 10netops: Move netflow data to Eventgate Analytics - https://phabricator.wikimedia.org/T248865 (10elukey) @Ottomata first n00b question - I am trying to think about where to add the netflow schema to the secondary repository, and I have some doubts about the dir structure. Should i... [10:38:24] * elukey lunch! [10:40:17] 10Analytics, 10Analytics-Kanban, 10Operations, 10vm-requests, 10Patch-For-Review: Create archiva1002 as replacement of archiva1001 - https://phabricator.wikimedia.org/T254890 (10jbond) p:05Triage→03Medium [10:41:46] 10Analytics-Data-Quality, 10QuickSurveys, 10Research, 10WMDE-Technical-Wishes-Team, and 3 others: Remove Do Not Track support for QuickSurveys - https://phabricator.wikimedia.org/T254224 (10awight) >>! In T254224#6209918, @WMDE-Fisch wrote: > Not as a blocker but I just wanted to mention that we currently... [10:42:40] 10Analytics-Data-Quality, 10QuickSurveys, 10Research, 10WMDE-Technical-Wishes-Team, and 3 others: Remove Do Not Track support for QuickSurveys - https://phabricator.wikimedia.org/T254224 (10awight) >>! In T254224#6210369, @awight wrote: > Thanks for pointing this out! I get it now--you already created the... [11:50:20] (03CR) 10Joal: "> Are jobs that use unzipping going to use this class explicitily? If so maybe we can add docs of how is it used?" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603590 (https://phabricator.wikimedia.org/T243241) (owner: 10Joal) [12:59:03] 10Analytics-Radar, 10Fundraising-Backlog, 10WMDE-Analytics-Engineering, 10WMDE-FUN-Team, 10WMDE-Fundraising-Tech: Find a better way for WMDE to get impression counts for their banners - https://phabricator.wikimedia.org/T243092 (10kai.nissen) @Nuria Sorry, that this has been sitting around for such a lon... [13:18:53] 10Analytics, 10Analytics-Kanban: Move Archiva to Debian Buster - https://phabricator.wikimedia.org/T252767 (10elukey) [13:35:37] (03PS1) 10Milimetric: [WIP] Clean up data flow as pertains to state [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/604387 [13:53:23] 10Analytics, 10Analytics-Kanban, 10Operations, 10observability, 10Patch-For-Review: systemd::syslog conf should use :programname equals instead of startswith - https://phabricator.wikimedia.org/T251606 (10Ottomata) Let's! [13:58:58] 10Analytics, 10Operations, 10netops: Move netflow data to Eventgate Analytics - https://phabricator.wikimedia.org/T248865 (10Ottomata) Yeah we don't have a great convention for namepacing. For analytics/instrumentation schemas, we decided to keep things simple and keep the hierarchy mostly flat, e.g. analyt... [14:10:08] (03CR) 10Joal: [C: 03+1] "Looks good - Did a quick check, I assume all the comments I made in previous patch have been addressed :)" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603591 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [14:13:04] 10Analytics, 10Operations, 10netops: Move netflow data to Eventgate Analytics - https://phabricator.wikimedia.org/T248865 (10elukey) @ayounsi any suggestion? `netflow/flow/something` ? [14:20:17] (03CR) 10Joal: "One last thing about the spark session created then ready IMO" (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/602475 (https://phabricator.wikimedia.org/T238230) (owner: 10Ottomata) [14:21:02] ottomata: when you want I'd happily discuss the location of event-related java code - I wonder if refinery-core is the best place, or if we should have a refinery-event module [14:22:06] joal: ya dunno, i also will eventually want to ingrate that with spark stuff [14:22:12] so you could easiliy get a spark schema using a stream name [14:22:21] elukey: what is a netflow? [14:22:27] what does the data in an event represent? [14:22:56] ottomata: for instance, given a stream-name, you retrieve the schema, and therefore a spark schema [14:24:01] ya [14:24:07] makes sense [14:24:21] could use that to automate spark streaming [14:24:51] its mostly that refinery-core was better than anywhere else [14:24:59] i'm not sure we are ready to make a new module just for this stuff [14:25:03] buuuut maybe later we can revisit that idea? [14:25:09] ottomata: Given that represent its own need (schema, stream, config, refine, streaming), I think a module could be beneficial - otherwise we endup as now we some code in refienry-core, some in refinery-job, and possibly some in refinery-spark [14:25:11] ottomata: it should be a flow of packets/bytes from Clients to our routers and vice-versa, but precisely I am a little ignorant, this is why I proposed something like netflow/flow/something [14:25:36] netflow is the name of the "protocol", and every event should be a data related to a flow [14:27:56] when is the event emitted? [14:28:00] what causes it to be emitted? [14:28:47] ottomata: pmacct pulls data from the routers and aggregate flows IIRC, and it sends those aggregated flows of bytes from IP A to IP B to Kafka [14:29:29] one use case is to monitor traffic flowing from one BGP AS to us or vice-versa [14:31:42] ottomata: changing subject - what do you think about something like https://phabricator.wikimedia.org/tag/analytics-cluster/ ? [14:32:26] so tagging the "ops" tasks with analytics-cluster, restoring its glorious name, and have a clear quarter by quarter schedule? [14:32:40] elukey: so each netflow event represents traffic from one Ip to another? [14:33:01] I think so [14:33:08] and it is aggregate info [14:33:38] yes, but I don't know the details [14:33:39] elukey: am all for it! [14:33:42] re analytics-cluster [14:33:52] super, will start doing something then :) [14:35:08] network/connection_summary ? [14:35:13] netflow/connection_summary? [14:35:20] network/netflow_summary? [14:35:25] dunno about summary, just an idea [14:35:53] ya hm. [14:36:00] joal [14:36:03] refinery-event [14:36:10] depends on refinery-core and refinery-spark [14:36:10] ? [14:36:23] would we move EventSchemaLoader stuff there, but leave JsonSchema loader stuff in core? [14:37:12] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban: Add new kafka brokers kafka-jumbo100[789] to the jumbo-eqiad Kafka cluster - https://phabricator.wikimedia.org/T252675 (10elukey) [14:37:13] ottomata: possibly why not, and we would move there event related stuff (refine, stream-config, etc) [14:37:52] oh [14:37:52] Sounds nice ottomata - JsonLoader is core indeed, the rest more event oriented [14:37:56] 10Analytics: rev_user and rev_user_text == NULL in wmf_raw.mediawiki_revision - https://phabricator.wikimedia.org/T254835 (10elukey) [14:38:00] not Refine, that is a job [14:38:08] makes sense ottomata [14:38:20] hmmmmm, i mean ok, i don't like more modules [14:38:31] do we have to modify the symlink jenkins job? [14:38:38] We would [14:38:49] Let's see what others think [14:38:50] 10Analytics, 10Analytics-Wikistats: Add proper trend numbers to wikistats metrics - https://phabricator.wikimedia.org/T251813 (10elukey) [14:38:54] i guess, i'm not opposed and i agree the organizaion makes sense [14:39:02] but i'm not sure what the gain for us is [14:39:02] 10Analytics, 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Package dictionaries better for ORES models - https://phabricator.wikimedia.org/T217343 (10elukey) [14:39:08] hmm [14:39:11] 10Analytics: Combine Hive Year / Month / Day / Hour partitions into ISO date string - https://phabricator.wikimedia.org/T177097 (10elukey) [14:39:14] i guess it allows e.g. EventStream to use refinery-spark. [14:39:15] hm [14:39:26] 10Analytics, 10Analytics-Kanban: Sanitize pageview_hourly - https://phabricator.wikimedia.org/T114675 (10elukey) [14:39:34] and we don't want to put EventSTream in refinery-spark or refinery job [14:39:36] 10Analytics: Refactor webrequest_source partitions and oozie jobs - https://phabricator.wikimedia.org/T116387 (10elukey) [14:39:43] hm [14:40:29] ottomata: it helps separate code, instead of having bits and pieces in many places [14:40:36] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Move Matomo to Debian Buster - https://phabricator.wikimedia.org/T252740 (10elukey) [14:40:47] ottomata: But this is still early, we can continue as is and make decision later [14:41:05] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban: Move Archiva to Debian Buster - https://phabricator.wikimedia.org/T252767 (10elukey) [14:41:22] 10Analytics, 10Analytics-Cluster, 10Cassandra, 10User-Elukey: Cassandra3 migration plan proposal - https://phabricator.wikimedia.org/T249756 (10elukey) [14:41:23] joal: yeah, but separating code in java isn't just separating code, it means more binary jars to juggle [14:41:32] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban: Upgrade AMD ROCm to latest upstream - https://phabricator.wikimedia.org/T247082 (10elukey) [14:41:42] but i agree that using spark in EventStream class will be good [14:41:45] so maybe it is needed [14:41:51] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, 10Product-Analytics, 10Epic: Spark sessions can provision kerberos tickets in a more predictable manner - https://phabricator.wikimedia.org/T246132 (10elukey) [14:41:53] unless we were to move all of this to refiner-job [14:41:53] 10Analytics: rev_user and rev_user_text == NULL in wmf_raw.mediawiki_revision - https://phabricator.wikimedia.org/T254835 (10JAllemandou) @diego : The actor table is in the above list: `mediawiki_private_actor`. The table is named `private` as it comes from the production DB. The labsDB `actor` view is using joi... [14:42:03] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, 10Operations: Create a profile to standardize the deployment of JVM packages and configurations - https://phabricator.wikimedia.org/T253553 (10elukey) [14:42:16] 10Analytics, 10Analytics-Cluster, 10Operations, 10ops-eqiad, 10User-Elukey: replace onboard NIC in kafka-jumbo100[1-6] - https://phabricator.wikimedia.org/T236327 (10elukey) [14:42:24] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, 10Analytics-SWAP, and 2 others: pip not accessible in new SWAP virtual environments - https://phabricator.wikimedia.org/T247752 (10elukey) [14:42:44] 10Analytics, 10Analytics-Cluster, 10Operations, 10netops: Move netflow data to Eventgate Analytics - https://phabricator.wikimedia.org/T248865 (10elukey) [14:42:46] ottomata: My assumption is that this code will be needed for more and more jobs, possibly batch and/or streaming [14:42:54] 10Analytics, 10Analytics-Cluster, 10Patch-For-Review: Decomission notebook hosts - https://phabricator.wikimedia.org/T249752 (10elukey) [14:42:59] 10Analytics, 10Analytics-Cluster, 10User-Elukey: Upgrade Druid to its latest upstream version (currently 0.18.1) - https://phabricator.wikimedia.org/T244482 (10elukey) [14:43:12] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10elukey) [14:43:16] ottomata: We might need in any case to rethink code organization if job number gets higher (or possibly just grow the number of packages) [14:44:33] ahahah sorry for the spam [14:44:47] (03CR) 10Ottomata: [C: 03+2] "Ok let's merge this to simplify other patch rebases." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603591 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [14:46:11] (03CR) 10Ottomata: Add EvolveHiveTable CLI tool to manually evolve Hive tables from JSONSchemas (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/602475 (https://phabricator.wikimedia.org/T238230) (owner: 10Ottomata) [14:46:21] (03PS6) 10Ottomata: Add EvolveHiveTable CLI tool to manually evolve Hive tables from JSONSchemas [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/602475 (https://phabricator.wikimedia.org/T238230) [14:47:41] (03CR) 10Joal: "Again one thing: Are we sure that the SparkSession is created in local (as it is not made explicitly)?" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/602475 (https://phabricator.wikimedia.org/T238230) (owner: 10Ottomata) [14:47:46] 10Analytics, 10Analytics-Cluster, 10Operations, 10netops: Move netflow data to Eventgate Analytics - https://phabricator.wikimedia.org/T248865 (10Ottomata) Ya if possible, the schema should be named and modeled after what the event represents. In this case it sounds like it is something like a 'network co... [14:49:41] 10Analytics, 10Analytics-Cluster: Upgrade schema[12]00[12] to Debian Buster - https://phabricator.wikimedia.org/T255026 (10elukey) [14:51:14] 10Analytics-Cluster, 10Analytics-Radar, 10Operations, 10observability: Move kafkamon hosts to Debian Buster - https://phabricator.wikimedia.org/T252773 (10elukey) [14:51:58] 10Analytics, 10Analytics-Cluster: Move the stat1004-6-7 hosts to Debian Buster - https://phabricator.wikimedia.org/T255028 (10elukey) [14:55:49] joal: you know, i think i would like to have all the non event patches merged for train [14:55:57] it will help me with the eventlogging migration [14:56:03] the event ingestion is another task [14:56:19] so, all the non WIP patches [14:56:24] https://gerrit.wikimedia.org/r/q/owner:ottomata+project:analytics%252Frefinery%252Fsource+status:open [14:56:26] think we can do that? [14:58:37] ottomata: The only one that is not yet reviewed is the [WIP] one - and it's big [15:00:47] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Technical contributors emerging communities metric definition, thick data - https://phabricator.wikimedia.org/T250284 (10Nuria) Notes from meeting: Let's try to understand what the bots are doing in detail to classify what are the types of contributions.... [15:00:52] ya WIp is not ready [15:03:37] ottomata: then we should probably be able to merge fast [15:11:36] 10Analytics: rev_user and rev_user_text == NULL in wmf_raw.mediawiki_revision - https://phabricator.wikimedia.org/T254835 (10diego) Thanks @JAllemandou [15:11:56] 10Analytics: rev_user and rev_user_text == NULL in wmf_raw.mediawiki_revision - https://phabricator.wikimedia.org/T254835 (10diego) 05Open→03Resolved [15:36:09] 10Analytics, 10Analytics-Cluster, 10Operations, 10netops: Move netflow data to Eventgate Analytics - https://phabricator.wikimedia.org/T248865 (10elukey) `netflow/flow/record` or `netflow/flow/observe` could be ok? [15:44:25] (03CR) 10Joal: [C: 03+2] "Forget my last comment - Merging" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/602475 (https://phabricator.wikimedia.org/T238230) (owner: 10Ottomata) [15:44:53] (03PS7) 10Ottomata: Add EvolveHiveTable CLI tool to manually evolve Hive tables from JSONSchemas [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/602475 (https://phabricator.wikimedia.org/T238230) [15:45:43] (03CR) 10Joal: [C: 03+2] "Same here - let's not wait more" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/602463 (owner: 10Ottomata) [15:46:04] (03CR) 10Joal: [C: 03+2] "Merging for dpeloy" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/601865 (owner: 10Ottomata) [15:50:56] (03Merged) 10jenkins-bot: DataFrameToHive - drop partition before writing output data [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/602463 (owner: 10Ottomata) [15:51:12] (03Merged) 10jenkins-bot: Refine - Make event transform functions smarter about choosing which possible column to use [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/601865 (owner: 10Ottomata) [15:51:42] 10Analytics, 10Analytics-Cluster, 10Operations, 10netops: Move netflow data to Eventgate Analytics - https://phabricator.wikimedia.org/T248865 (10Ottomata) I'd guess I'd still ask what is a "netflow"? Or a "netflow/flow"? I guess if you could defined 'netflow' as a noun in the description of the schema,... [15:58:32] 10Analytics, 10Analytics-Cluster, 10Operations, 10netops: Move netflow data to Eventgate Analytics - https://phabricator.wikimedia.org/T248865 (10Ottomata) observe could be ok too, sorry didn't mean to make observation sound better or worse. netflow/observe event sounds a little weird but does seem consis... [15:59:22] 10Analytics, 10Analytics-Cluster, 10Operations, 10netops: Move netflow data to Eventgate Analytics - https://phabricator.wikimedia.org/T248865 (10Ottomata) ALSO these are just ideas and thoughts! Schemas in secondary repo SHOULD require less bikeshedding than those in primary :) [16:01:15] 10Analytics, 10Analytics-Cluster, 10Operations, 10netops: Move netflow data to Eventgate Analytics - https://phabricator.wikimedia.org/T248865 (10elukey) Netflow in theory is the name of the technology/protocol (see https://tools.ietf.org/html/rfc3954), and IIUC it defines a "flow" as the bytes/packets exc... [16:04:44] 10Analytics, 10Analytics-Cluster: [Spike] Explore goblin as an alternative to camus - https://phabricator.wikimedia.org/T252560 (10elukey) [16:08:14] ottomata: all merged (I double checked) [16:08:28] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban: Move Archiva to Debian Buster - https://phabricator.wikimedia.org/T252767 (10elukey) Some notes from the other migration: - Remember to turn on `profile::archiva::proxy::only_localhost` when applying the role archiva to archiva1002, so the new installati... [16:13:43] 10Analytics, 10Analytics-Cluster, 10Operations, 10netops: Move netflow data to Eventgate Analytics - https://phabricator.wikimedia.org/T248865 (10Ottomata) Ah, makes more sense! Great. If pmacct is aggregating, perhaps summary is good in the name? [16:14:07] joal: thank youUuuu [16:16:54] 10Analytics, 10Operations, 10SRE-Access-Requests, 10Product-Analytics (Kanban): Creation of a new POSIX group and system user for the Product Analytics team - https://phabricator.wikimedia.org/T255039 (10elukey) [16:16:57] ottomata: fyi, there's some new high-volume topic going through eventgate: all MW purges. https://grafana.wikimedia.org/d/ePFPOkqiz/eventgate?panelId=54&fullscreen&orgId=1&refresh=1m [16:18:13] bearloga: o/ - I created T255039 since it is a sudo access request formally, so there are some rules before being able to merge, will try to follow up with John [16:18:13] T255039: Creation of a new POSIX group and system user for the Product Analytics team - https://phabricator.wikimedia.org/T255039 [16:18:36] elukey: thank you!! [16:19:14] nice Pchelolo ! [16:19:21] very cool [16:26:39] mforns: o/ = forgot to ping you but the issue that you had with monitor refine logs not present on an-launcher should be solved, andrew merged today a puppet change for it [16:27:14] ah! ok, thanks! it wasn't my imagination then :) [16:27:48] nono I checked and it was what Andrew already found, and he fixed it today :) [16:50:57] ottomata: I'm about to deploy refinery-source and there's several refine things to deploy, do I need to do anything special? or just bump up the refinery_jar_version in puppet? [16:51:15] see: https://etherpad.wikimedia.org/p/analytics-weekly-train [16:52:06] mforns: i will do the stuff in puppet [16:52:20] i want to do that next week when I am ready to think about all that and watch it [16:52:26] for refimme [16:52:28] for refine [16:52:43] mforns: when you are done train, got a min for some more java code brain bounce? [16:52:48] want to think about what do to about event service urls [17:01:50] ottomata: yes sure, but you mean I should freeze deployment until we discuss? [17:02:09] nono [17:02:12] ok ok [17:02:13] deploy away [17:02:17] the event stuff is not merged yet [17:02:21] wanna bc now? [17:02:23] iongestion stuff [17:02:24] sure! [17:02:27] ok [17:03:07] google meet is slow [17:04:45] 10Analytics-Data-Quality, 10QuickSurveys, 10Research, 10WMDE-Technical-Wishes-Team, and 3 others: Remove Do Not Track support for QuickSurveys - https://phabricator.wikimedia.org/T254224 (10ovasileva) [17:12:31] Anyone here know anything about "WDQS-Streaming-Updater-POC"? We're on a trajectory to run out of space on Graphite due to all the new metrics being pushed: https://w.wiki/TeN [17:12:50] shdubsh: it's me [17:13:18] will stop sending metrics sorry about that [17:14:47] dcausse: do you need metrics for this POC work or is it easy enough to simply turn them off? [17:15:34] shdubsh: I need a couple of them, will find a way to filter them [17:21:27] 10Analytics, 10Analytics-Cluster, 10Operations, 10netops: Move netflow data to Eventgate Analytics - https://phabricator.wikimedia.org/T248865 (10Nuria) analytics/network/netflow/flowset? analytics/netflow/flowset? Flowset is how an "event" is called on the protocol linked by @elukey above [17:24:03] 10Analytics-Radar, 10Fundraising-Backlog, 10WMDE-Analytics-Engineering, 10WMDE-FUN-Team, 10WMDE-Fundraising-Tech: Find a better way for WMDE to get impression counts for their banners - https://phabricator.wikimedia.org/T243092 (10Nuria) >Currently, the data is published to the respective hive table with... [17:24:10] 10Analytics: Many new metrics in Graphite for WDQS-Streaming-Updater-POC - https://phabricator.wikimedia.org/T255044 (10colewhite) [17:24:26] dcausse: created https://phabricator.wikimedia.org/T255044 for tracking [17:24:47] 10Analytics, 10Analytics-Cluster, 10Operations, 10netops: Move netflow data to Eventgate Analytics - https://phabricator.wikimedia.org/T248865 (10Ottomata) I don't think this needs to go in 'analytics', but flowset sounds nice if it is accurate. [17:24:50] shdubsh: thanks, if it's easy on your side you can delete them [17:24:57] to reclaim this space [17:24:59] ack! [17:25:26] I totally understimated the size of these metrics :( [17:26:32] hmmmm mforns another idea [17:26:38] i could just use the convention we have [17:26:43] aha [17:26:50] the event service names always match the urls [17:26:54] they don't have to [17:26:55] but so far they do [17:27:00] aha [17:27:11] 10Analytics, 10Analytics-Cluster, 10Operations, 10netops: Move netflow data to Eventgate Analytics - https://phabricator.wikimedia.org/T248865 (10Nuria) network/netflow/flowset? [17:27:19] eventgate-main -> eventgate-main.discovery.wmnet AND/OR eventgate-main.svc.codfw.wmnet [17:27:30] when you say "the convention" makes me think about hannabi xD [17:27:45] I see [17:28:14] haha [17:29:14] but how do you know the second part of the uri? like the discovery.wmnet/svc.codfw.wmnet? [17:29:24] i'd infer it [17:30:46] getEventServiceUri(name) -> name.discovery.wmnet [17:30:46] getEventServiceUri(name, datacenter) -> name.svc.datacenter.wmnet [17:31:12] aha [17:36:11] dcausse: let me know (or on task) when your mitigations are in place. if I delete them before, they will just be recreated again. [17:42:38] AHHH no mforns the ports vary [17:42:39] GRRRR [17:42:48] :((((( [17:42:59] 10Analytics-Radar, 10Product-Analytics: /srv/published should be structured similarly, have identical README across stat hosts describing said structure - https://phabricator.wikimedia.org/T254189 (10Milimetric) Hm, if you want to do this in puppet, and not manually, then we can help, but I want to first doubl... [17:59:32] * elukey off! [17:59:48] bye3 elukey [18:16:50] joal: s [18:16:53] à bientôt! [18:16:56] :) [18:17:01] see you tomorrow :) [19:21:58] team, with all the meetings today, I didn't start the deployment train yet. I think I will leave it for tomorrow before standup [19:52:54] (03PS6) 10Ottomata: [WIP] Add EventStreamConfig and EventStream classes [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603582 (https://phabricator.wikimedia.org/T251609) [19:53:24] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Add EventStreamConfig and EventStream classes [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603582 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [19:56:51] 10Analytics, 10Analytics-Cluster, 10Operations, 10ops-eqiad, 10User-Elukey: replace onboard NIC in kafka-jumbo100[1-6] - https://phabricator.wikimedia.org/T236327 (10Jclark-ctr) @elukey I am on site every tuesday and thursday. usually arrive at 9:00am est message me on irc to workout a schedule that... [20:02:02] * nshahquinn a-team: I'm on stat1008, and when I run the command ` kafkacat -C -b kafka-jumbo1001 -t eventlogging_KaiOSFirstRun`, I get the error `% ERROR: Topic eventlogging_KaiOSFirstRun error: Broker: Leader not available`, even though that topic is listed with a leader when I run `kafkacat -L`. Any idea what's going on? [20:02:27] nshahquinn: kafka-jumbo1001.eqiad.wmnet [20:02:27] ? [20:02:45] actually [20:02:47] kafka-jumbo1001.eqiad.wmnet:9092 [20:03:32] ottomata: ah, duh, thanks [20:07:14] ottomata: FYI, it actually works without the port number [20:07:20] oh didn't know that [20:07:20] cool [20:09:17] java is the worst [20:09:30] i have just spent 15 minutes trying to figure out how to make an http post with a json body [20:09:36] still figuring itout... [20:17:13] * nshahquinn ottomata: okay, here's another one. If I try to consume from a topic, that doesn't exist, I get an error. Makes sense. BUT then the non-existent topic shows up in the metadata list that I get when running `kafkacat -L`. Is that how it's supposed to work? [20:17:43] For example: [20:17:59] https://www.irccloud.com/pastebin/lv87uY7m/ [20:22:44] nshahquinn: ya auto topic creation is on right now [20:22:55] and our versioj doesn't have a way to prevent consumers from creating a topic [20:23:08] so whenever you try to produce or consume a topic that doesn't exist, kafka will create it [20:23:28] * nshahquinn hah okay, well, I just created 5 new topics or so by mistyping the one I was trying to get...as long as that's okay :D [20:23:31] haha [20:23:36] theres lots of junk in there [20:23:37] its ok [20:23:45] * nshahquinn cool :) [20:23:46] it'd be betteer not to do that, but you aren't the only source of junk [20:23:55] and until we get a newer version, it'll probably keep happening [20:23:58] I'll try my best [20:24:24] * nshahquinn thanks for tuning in to today's episode of Neil's Adventures with Kafkacat! [20:25:54] * nshahquinn why do all my messages look like I typed `/me message`? not important, but weird [20:26:27] * nshahquinn actually, only some of my messages...is it something with my client (IRC Cloud)? [20:26:33] test [20:26:45] ohhh, I see, when I hit command + enter [20:26:57] I'm used to doing that in Slack now [20:27:20] Neil's Adventures with IRC [20:32:34] * addshore test [20:32:36] ooooh [20:32:42] i just learnt something new [20:37:19] test [20:37:26] test [20:37:53] * delphine test [20:39:08] :D [21:36:35] ottomata: I tried to do that in Java on Android a few years ago and I basically gave up after a day. It's crazy that it's both super standard and ordered and gross but at the same time lacks basic things like a good standard library to do this. [21:36:50] just for fun I'm going to look this up for C#, I don't remember how to do it [21:38:42] ok, found a bunch of different ways to do it, with a great tutorial on the official docs and a bunch of convenient third-party libraries: https://docs.microsoft.com/en-us/dotnet/framework/network-programming/how-to-send-data-using-the-webrequest-class [21:47:06] 10Analytics, 10Analytics-Kanban: Spike, see how easy/hard is to scoop all tables from Eventlogging log database - https://phabricator.wikimedia.org/T250709 (10Nuria) Some comments: We do not need to scoop the following tables as that data exists on events database and mediawiki_history, let's remove them from... [21:55:56] 10Analytics, 10Analytics-Kanban: Spike, see how easy/hard is to scoop all tables from Eventlogging log database - https://phabricator.wikimedia.org/T250709 (10Nuria) Potential issues: ` Empty webhost MobileWikiAppEdit_17837072 +-------+------------+------------------+-----------------------------+ |webhos... [21:58:28] 10Analytics, 10Analytics-Kanban: Spike, see how easy/hard is to scoop all tables from Eventlogging log database - https://phabricator.wikimedia.org/T250709 (10Nuria) Empty event_token event_userBuckets ` ServerSideAccountCreation_17719237 +-----------------+--------------------+----+-----------+-----------... [22:08:00] 10Analytics, 10Analytics-Kanban: Spike, see how easy/hard is to scoop all tables from Eventlogging log database - https://phabricator.wikimedia.org/T250709 (10Nuria) PageContentSaveComplete_5588433 userAgent column is null that might be actually correct but let's check it out [22:09:48] 10Analytics: Grant not able to access superset - https://phabricator.wikimedia.org/T253281 (10Nuria) 05Open→03Resolved [22:13:20] 10Analytics, 10Analytics-Kanban: Spike, see how easy/hard is to scoop all tables from Eventlogging log database - https://phabricator.wikimedia.org/T250709 (10Nuria) MobileWikiAppArticleSuggestions_10590869_15423246 event_pageTitle is null [22:14:25] 10Analytics, 10Analytics-Kanban: Spike, see how easy/hard is to scoop all tables from Eventlogging log database - https://phabricator.wikimedia.org/T250709 (10Nuria) I just did some spot checking, we probably need to take a look at these to see if these are issues and if so, if they can be fixed globally (cc... [22:21:07] milimetric: aye yeah it looks basically the same in java then too [22:22:05] milimetric: it might be better in java 11, i skipped the docs that had some tips for that [22:37:13] 10Analytics-Radar, 10Core Platform Team, 10Dumps-Generation: HTML Dumps - June/2020 - https://phabricator.wikimedia.org/T254275 (10Jseddon) Thanks for the questions and queries, much appreciated and the team will try to give good responses ASAP. Thanks all for your patience [23:00:42] Right but my point is like when you search for this stuff in Java you get two doctoral theses that disagree