[05:09:39] 10Analytics, 10DBA, 10Data-Services, 10cloud-services-team, and 2 others: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Marostegui) After merging puppet I ran the following on the 4 labsdb hosts to remove the temporary index on the hiwikisource view for ar... [05:48:11] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban: Archive data on eventlogging MySQL to analytics replica before decomisioning - https://phabricator.wikimedia.org/T231858 (10Marostegui) I think we should explore what Nuria suggested, trying to drop as much as possible before deciding where to archiv... [07:17:25] joal: bonjour! [07:17:28] all set for Berlin :) [07:17:37] \o/ ! Bonjour elukey :) [07:18:52] I'll stay a couple of days more to visit [07:19:49] ok :) [08:02:30] TIL https://docs.oracle.com/en/java/javase/11/gctuning/z-garbage-collector1.html#GUID-A5A42691-095E-47BA-B6DC-FB4E5FAA43D0 [08:02:33] (with Java 11) [08:03:08] moritzm: https://blog.cloudera.com/cdh6-3-hbase-g1-gc-tuning-with-jdk11/?ite=34304&ito=2155&itq=05910304-74ad-47ea-8851-c0081071cc57&itx%5Bidio%5D=97602 looks promising, cloudera made a blogpost about CDH6.3 and Java 11 usage and tuning [08:41:45] elukey: from https://www.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_java_requirements.html, it looks like we can go for openJDK-11 with CDH 6.3 :) [08:42:50] yes in theory 6.3 is the first of their version with support for 11, I hope not experimental :) [08:42:57] so do I :) [08:43:07] elukey: shall we test that? :) [08:43:30] joal: it is not super easy :) [08:43:39] I can imagine that [08:43:46] elukey: maybe labs at first? [08:44:12] one thing at the time, first kerberos then Buster with CDH 5.X and java-8 then CDH 6 with java-8 in my opinion.. Then java 11 :) [08:44:34] ok, ok :) [08:44:52] Maybe Druid and Zookeeper will need to be switched to 11 as well [08:44:54] not sure [08:45:34] for this fiscal if we manage to move to Buster and test CDH6 it will be a ton of progress in my opinion :) [08:45:44] with the big IF of following or not CDH [08:46:09] I hope to get some answers in Berlin [08:46:50] indeed elukey [08:54:41] Zookeeper as packaged in Debian already uses 11, so that should be fine [08:58:39] yeah I meant all clients using java-based libs to run the same jvm [09:29:22] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Move refinery to hive 2 actions - https://phabricator.wikimedia.org/T227257 (10elukey) [09:31:06] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Move refinery to hive 2 actions - https://phabricator.wikimedia.org/T227257 (10elukey) To keep archives happy, Joseph also followed up with: https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/531682/ https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+... [10:29:37] (03PS1) 10Elukey: util.py: return utf-8 text string for stdout/stderr [analytics/refinery] - 10https://gerrit.wikimedia.org/r/538235 (https://phabricator.wikimedia.org/T204735) [10:42:21] (03PS2) 10Elukey: util.py: return utf-8 text string for stdout/stderr [analytics/refinery] - 10https://gerrit.wikimedia.org/r/538235 (https://phabricator.wikimedia.org/T204735) [10:42:43] (03PS3) 10Elukey: util.py: return text strings for stdout/stderr [analytics/refinery] - 10https://gerrit.wikimedia.org/r/538235 (https://phabricator.wikimedia.org/T204735) [10:57:08] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Move the Analytics Refinery to Python 3 - https://phabricator.wikimedia.org/T204735 (10elukey) After checking the code I think that the problem is in the util.sh() function, that with python2 returns a text string meanwhile with Python3 a bytes string. Tes... [11:37:31] * elukey lunch + errand [12:02:27] PROBLEM - Check the last execution of refinery-import-page-history-dumps on stat1007 is CRITICAL: connect to address 10.64.21.118 port 5666: Connection refused https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [12:06:23] PROBLEM - Check the last execution of archive-maxmind-geoip-database on stat1007 is CRITICAL: connect to address 10.64.21.118 port 5666: Connection refused https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [12:06:51] PROBLEM - Check the last execution of reportupdater-interlanguage on stat1007 is CRITICAL: connect to address 10.64.21.118 port 5666: Connection refused https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [12:12:59] RECOVERY - Check the last execution of refinery-import-page-history-dumps on stat1007 is OK: OK: Status of the systemd unit refinery-import-page-history-dumps https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [12:16:57] RECOVERY - Check the last execution of archive-maxmind-geoip-database on stat1007 is OK: OK: Status of the systemd unit archive-maxmind-geoip-database https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [12:17:27] RECOVERY - Check the last execution of reportupdater-interlanguage on stat1007 is OK: OK: Status of the systemd unit reportupdater-interlanguage https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [13:05:45] (03CR) 10Nuria: [C: 03+2] util.py: return text strings for stdout/stderr [analytics/refinery] - 10https://gerrit.wikimedia.org/r/538235 (https://phabricator.wikimedia.org/T204735) (owner: 10Elukey) [13:22:03] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban: Archive data on eventlogging MySQL to analytics replica before decomisioning - https://phabricator.wikimedia.org/T231858 (10elukey) Some data: ` elukey@db1107:/srv/sqldata$ ls _* | cut -d "_" -f 2 | sort | uniq -c 5 2198 log 16 otto... [13:22:39] (03CR) 10Elukey: [V: 03+2] util.py: return text strings for stdout/stderr [analytics/refinery] - 10https://gerrit.wikimedia.org/r/538235 (https://phabricator.wikimedia.org/T204735) (owner: 10Elukey) [13:23:33] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Version analytics meta mysql database backup - https://phabricator.wikimedia.org/T231208 (10elukey) [13:28:34] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Allow all Analytics tools to work with Kerberos auth - https://phabricator.wikimedia.org/T226698 (10elukey) @mforns you have another use case that needs rsync from the HDFS mount right? [13:34:22] 10Analytics: Verify what Python 2 packages deployed to Analytics hosts are needed - https://phabricator.wikimedia.org/T204737 (10elukey) I think that the task can be closed, please re-open if necessary! [13:34:27] 10Analytics: Verify what Python 2 packages deployed to Analytics hosts are needed - https://phabricator.wikimedia.org/T204737 (10elukey) [13:34:45] 10Analytics, 10Analytics-Kanban: Verify what Python 2 packages deployed to Analytics hosts are needed - https://phabricator.wikimedia.org/T204737 (10elukey) [14:01:20] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Allow all Analytics tools to work with Kerberos auth - https://phabricator.wikimedia.org/T226698 (10mforns) @elukey Yes! MediaWiki history dumps need to be rsync'd from HDFS mount. [14:02:49] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Allow all Analytics tools to work with Kerberos auth - https://phabricator.wikimedia.org/T226698 (10elukey) >>! In T226698#5509314, @mforns wrote: > @elukey > Yes! MediaWiki history dumps need to be rsync'd from HDFS mount. Can you tell m... [14:18:09] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Allow all Analytics tools to work with Kerberos auth - https://phabricator.wikimedia.org/T226698 (10mforns) @elukey I'm not sure about the underlying magic, but what I've seen in other similar cases of data sets that need rsync and publica... [14:18:17] heyyy teamm [14:19:18] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), 10Services (watching): Modern Event Platform: Schema Registry: Implementation - https://phabricator.wikimedia.org/T206789 (10Ottomata) [14:19:40] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), 10Core Platform Team Workboards (Clinic Duty Team): Develop a library for JSON schema backwards incompatibility detection - https://phabricator.wikimedia.org/T206889 (10Ottomata) @Pchelolo should we close? [14:45:35] 10Analytics, 10Research: Parse wikidumps and extract redirect information for 1 small wiki, romanian - https://phabricator.wikimedia.org/T232123 (10MGerlach) @JAllemandou I came up with a first solution on spark (see attached notebooks; I ran this on the notebook-server). This creates a dataframe with all rev... [15:08:44] 10Analytics, 10Analytics-EventLogging, 10Better Use Of Data, 10EventBus, and 5 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10Ottomata) It sounds like the Druid use case needs a little more thought. I still think this is a good idea, but let'... [15:09:39] 10Analytics, 10CirrusSearch, 10Discovery, 10Discovery-Search: Ingest cirrussearchrequest data into druid - https://phabricator.wikimedia.org/T218347 (10Ottomata) Actually blocked on {T208589}. [15:12:05] 10Analytics, 10Analytics-EventLogging, 10Better Use Of Data, 10EventBus, and 5 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10Ottomata) @Pchelolo any thoughts? [15:12:58] 10Analytics, 10EventBus, 10Product-Analytics: Review draft Modern Event Platform schema guidelines - https://phabricator.wikimedia.org/T233329 (10Ottomata) Working on 'guidelines' here: https://wikitech.wikimedia.org/wiki/Event_Platform/Schemas/Guidelines [15:21:23] 10Analytics, 10Analytics-EventLogging, 10Better Use Of Data, 10EventBus, and 5 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10Ottomata) FYI Schema and Design Guideline documentation WIP here: - https://wikitech.wikimedia.org/wiki/Event_Platf... [15:21:31] 10Analytics, 10EventBus: Clean up descriptions of fields in included common schemas in mediawiki/event-schemas repository - https://phabricator.wikimedia.org/T233057 (10Ottomata) [15:22:30] helloooo joal any chance you're there? [15:24:54] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), 10Core Platform Team Workboards (Clinic Duty Team): Develop a library for JSON schema backwards incompatibility detection - https://phabricator.wikimedia.org/T206889 (10Pchelolo) 05Stalled→03Resolved [15:24:58] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), and 2 others: CI Support for Schema Registry - https://phabricator.wikimedia.org/T206814 (10Pchelolo) [15:25:57] mforns: hiii do you have any recollection of aqs returning value set as "long" as strings in the json? [15:26:17] hi! [15:26:31] what do you mean, long types returned as stings? [15:26:36] mforns: yes [15:26:53] mmmm, doesn't ring a bell [15:27:42] :[ [15:30:41] 10Analytics, 10Analytics-EventLogging, 10Better Use Of Data, 10EventBus, and 5 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10Pchelolo) Multiple annotations directly in definition looks better. [15:49:21] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), 10Services (watching): Figure out how to $ref common schema across schema repositorise - https://phabricator.wikimedia.org/T233432 (10Ottomata) [15:49:30] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), 10Services (watching): Figure out how to $ref common schema across schema repositories - https://phabricator.wikimedia.org/T233432 (10Ottomata) [15:51:20] elukey: sorry to bother you with this and feel free to ignore if you're busy with more pressing stuff, but cassandra in beta seems ded [15:51:34] https://www.irccloud.com/pastebin/Vf92noHv/ [15:59:46] fdans: ahhh lovely, let me check [16:03:56] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 3 others: CI Support for Schema Registry - https://phabricator.wikimedia.org/T206814 (10Ottomata) [16:04:09] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Better Use Of Data, and 6 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10Ottomata) [16:04:14] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review: Clean up descriptions of fields in included common schemas in mediawiki/event-schemas repository - https://phabricator.wikimedia.org/T233057 (10Ottomata) [16:05:19] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 2 others: Figure out how to $ref common schema across schema repositories - https://phabricator.wikimedia.org/T233432 (10Ottomata) [16:05:45] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review: Clean up descriptions of fields in included common schemas in mediawiki/event-schemas repository - https://phabricator.wikimedia.org/T233057 (10Ottomata) a:03Ottomata [16:16:42] fdans: cluster up :) [16:17:06] there was a change that I didn't know about, /etc/cassandra/service-enabled needs to be present to allow the cassandra systemd unit to start [16:43:17] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Better Use Of Data, and 6 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10Ottomata) @JAllemandou any thoughts? [17:00:04] mforns: am trying to think of a name for 'druid field type'. [17:00:17] don't want to say 'druid'; it is more generic and OLAPy [17:00:25] could be used for prometheus or other OLAP systems too [17:01:21] !log remove /var/lib/hadoop/j from analytics1045 due to a broken dis [17:01:23] mmmm [17:01:23] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:03:31] olap_type [17:03:31] ? [17:03:36] cube_type [17:04:04] etl_type [17:04:41] !log restart yarn/hdfs daemons on analytics1045 [17:04:43] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:04:47] this may cause some jobs to fail --^ [17:06:19] ottomata, aah! now I get your question [17:06:47] pro_cube [17:07:02] pro? [17:07:46] xD [17:07:52] cube_member [17:09:23] olap_unit [17:09:35] olap_field [17:11:22] ottomata, olap_dimension, olap_measure, out of ideas :] [17:11:28] olap_* yeah [17:11:35] but the annotation will loook like [17:11:44] olap_field: [time_measure, measure] [17:11:51] actually [17:11:57] it will probably be a csv string [17:12:02] olap_field: time_measure,measure [17:12:09] so it is kind of a 'type' [17:12:20] olap_type [17:12:24] cube_type [17:12:26] best i got so far [17:13:04] I see [17:13:34] olap_properties [17:14:14] hm [17:14:25] that's pretty good [17:14:27] olap_attrs [17:14:53] :D [17:16:23] 10Analytics, 10EventBus, 10Product-Analytics: Define how we vet code & data for ongoing, automated ingestion in Druid - https://phabricator.wikimedia.org/T210012 (10Ottomata) In {T214093} we are discussing how to annotate event schemas with druid ingestion information, to ease automated ingestion into druid.... [17:19:43] 10Analytics, 10EventBus, 10Product-Analytics: Define how we vet code & data for ongoing, automated ingestion in Druid - https://phabricator.wikimedia.org/T210012 (10Ottomata) > An issue with keeping the ingestion annotation in the event schema is that anyone who has merge rights to the analytics event schema... [17:44:21] 10Analytics, 10Analytics-Kanban: Move the Analytics Refinery to Python 3 - https://phabricator.wikimedia.org/T204735 (10mforns) @elukey Reviewed https://gerrit.wikimedia.org/r/538235 and the fix makes sense to me! There is a call to os.system returning a value, but I checked the python docs and it seems to be... [17:48:54] 10Analytics, 10CirrusSearch, 10Discovery, 10Discovery-Search: Ingest cirrussearchrequest data into druid - https://phabricator.wikimedia.org/T218347 (10Ottomata) [17:48:56] 10Analytics: [EventLoggingToDruid] Add support for ingesting subfields of map columns - https://phabricator.wikimedia.org/T208589 (10Ottomata) [17:52:18] 10Analytics, 10EventBus, 10Product-Analytics: Define how we vet code & data for ongoing, automated ingestion in Druid - https://phabricator.wikimedia.org/T210012 (10Ottomata) [17:52:27] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Better Use Of Data, and 6 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10Ottomata) [18:10:56] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10Readers-Web-Backlog (Tracking): Ingest data aggregate ReadingDepth data into Druid - https://phabricator.wikimedia.org/T205562 (10kzimmerman) [18:31:07] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Better Use Of Data, and 6 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10Ottomata) Hm @mforns, just considered how map types would look if we use annotations on schemas for privacy... [19:15:08] 10Analytics, 10EventBus, 10Product-Analytics: Review draft Modern Event Platform schema guidelines - https://phabricator.wikimedia.org/T233329 (10Neil_P._Quinn_WMF) >JSONSchema provides powerful data validation, but unlike Avro, it does not provide schema evolution. That is, each schema is distinct, and ther... [19:26:29] 10Analytics, 10EventBus, 10Product-Analytics: Review draft Modern Event Platform schema guidelines - https://phabricator.wikimedia.org/T233329 (10Neil_P._Quinn_WMF) And a few more comments here (because I accidentally saved that last comment too early): > jsonschema-tools is a NodeJS libary and CLI for mana... [19:31:43] 10Analytics, 10EventBus, 10Product-Analytics: Review draft Modern Event Platform schema guidelines - https://phabricator.wikimedia.org/T233329 (10Neil_P._Quinn_WMF) 05Open→03Resolved Okay, that's everything I can think of right now. Feel free to ping me again if you have other questions or new drafts you... [19:31:54] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Better Use Of Data, and 6 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10Neil_P._Quinn_WMF) [19:40:18] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Better Use Of Data, and 6 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10mforns) @Ottomata Hm indeed... Specifying sanitization for the whole map can be dangerous, because a non-se... [19:47:03] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Better Use Of Data, and 6 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10Ottomata) Hm, your example isn't quite right. geocoded_data is defined to be a map of string -> string, and... [20:04:18] 10Analytics, 10EventBus, 10Product-Analytics: Review draft Modern Event Platform schema guidelines - https://phabricator.wikimedia.org/T233329 (10Ottomata) Thanks for comments! > So...we can't reliably upgrade producer and consumer code? That seems bad; did we work around this limitation somehow? Read on!... [20:45:14] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 3 others: CI Support for Schema Registry - https://phabricator.wikimedia.org/T206814 (10Ottomata) [20:46:19] 10Analytics: Some event data (like the one that comes from mediawiki events such us revision create) should not get sanitized - https://phabricator.wikimedia.org/T217271 (10Ottomata) [20:46:24] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 3 others: CI Support for Schema Registry - https://phabricator.wikimedia.org/T206814 (10Ottomata) [20:50:36] 10Analytics, 10Analytics-EventLogging, 10Reading Epics (Analytics): Bulk/Batch event endpoint - https://phabricator.wikimedia.org/T166249 (10Ottomata) [20:50:44] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 3 others: Modern Event Platform: Stream Intake Service (EventGate): Implementation - https://phabricator.wikimedia.org/T206785 (10Ottomata) [20:51:18] 10Analytics, 10Analytics-EventLogging, 10Tracking-Neverending: Use draft 4 of JSON Schema specification - https://phabricator.wikimedia.org/T46809 (10Ottomata) [20:51:20] 10Analytics, 10Analytics-EventLogging: Update client-side event validator to support (at least) draft 3 of JSON Schema - https://phabricator.wikimedia.org/T182094 (10Ottomata) 05Open→03Declined Closing this; Modern Event Platform uses draft-07. Won't change for EventLogging. [20:52:58] 10Analytics, 10Analytics-EventLogging, 10Wikimedia-production-error: EventLogging-based extensions cause errors on test2.wikipedia.org - https://phabricator.wikimedia.org/T196309 (10Ottomata) We won't be fixing this as part of Modern Event Platform. Can test2wiki's config just be updated to use meta.wm.org? [22:18:55] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Better Use Of Data, and 6 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10Nuria) >Perhaps we should add them into the event schema, even if they are never set by event producers? The... [23:49:36] goddamn, looking at the barrage of mediarequest alerts