[06:03:09] morning! [06:03:17] fdans: really sorry for the pw change! :( [06:03:47] neilpquinn: sure! I followed https://wikitech.wikimedia.org/wiki/SWAP#Resetting_user_virtualenvs [06:15:52] joal: bonjour! I am ready for druid - https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/505062/ [06:43:43] I am rolling restart overlord and coordinators atm (to reduce heap size) [06:55:44] Morning elukey [06:57:52] bonjour :) [06:58:09] I waited to roll restart brokers and hostoricals joal, waiting for your green light [06:59:00] All good form me - monitoring metrics [06:59:44] all right! starting with historicals [07:14:09] historicals done! [07:14:18] unnoticeable :) [07:17:47] and brokers done! [07:18:27] druid1001 is the only one used and cache misses have of course an impact [07:27:17] just checked and memory freed by reducing the heap sizes went to page cache [07:27:20] goooood [07:28:21] Thanks elukey ) [08:52:14] brb [09:02:49] joal elukey in the end refinery source deployed successfully yesterday, should I wait to monday to deploy the cluster and restart the load job or you think it's ok to do it now? [09:13:08] if it is not super urgent I'd wait until monday (me and Joseph will be off though, you'll need to sync with Andrew) [09:16:56] elukey: yessir, will deploy on Monday [09:19:19] fdans: let's also wait for joal's opinion - if there are not huge changes we can also think to deploy today and complete the work [09:21:12] heya - I don't mind - The change is relatively small and involves restart webrequest (no big either) - However it requires close monitoring in that new pageview-titles should still be relatively coherent [09:27:18] joal: you mentioned that we would have to restart the job from march or did I misunderstand? [09:49:13] 10Analytics, 10Analytics-Data-Quality, 10Product-Analytics, 10Patch-For-Review: Some registered users have null values for event_user_text and event_user_text_historical in mediawiki_history - https://phabricator.wikimedia.org/T218463 (10JAllemandou) Confirmation of problem resolution in new test-datasourc... [09:50:29] 10Analytics, 10Analytics-Data-Quality, 10Product-Analytics, 10Patch-For-Review: Some registered users have null values for event_user_text and event_user_text_historical in mediawiki_history - https://phabricator.wikimedia.org/T218463 (10JAllemandou) Hi @Neil_P._Quinn_WMF, sorry for the big comment above -... [09:56:26] 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, 10Product-Analytics: mediawiki_history datasets have null user_text for IP edits - https://phabricator.wikimedia.org/T206883 (10JAllemandou) Confirmation of problem resolution in new test-datasource located at /user/joal/wmf/data/wmf/mediawiki/user... [09:58:39] 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Some registered users have null values for event_user_text and event_user_text_historical in mediawiki_history - https://phabricator.wikimedia.org/T218463 (10JAllemandou) [09:59:16] 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Some registered users have null values for event_user_text and event_user_text_historical in mediawiki_history - https://phabricator.wikimedia.org/T218463 (10JAllemandou) a:05mforns→03JAllemandou [10:00:43] fdans: didn't you guys make an etherpad with the deploy plan? [10:01:25] if not (or if it lacks info) we can work today on having a list of things to do/check on monday [10:01:33] that could be done even without Joseph [10:02:16] this could become a good practice for the chu chu train [10:02:51] fdans: sorry missed your ping - no need to backfill [10:02:53] that people changing things to refinery help to create a procedure for deployments, that whoever has the ops week will execute [10:03:04] elukey: there wasn't that much to do in the end after our conversation, so we didn't make one [10:03:30] it was release source - deploy cluster - restart load bundle [10:03:54] sure, what about the things to check to make sure that everything works fine? [10:04:11] my point is that usually we rely on Joseph doing them :D [10:04:29] the thing not to forget in that procedure is: monitor-restarted-jobs (if feasible, or put a reminder to do it when it runs if in long time) :) [10:04:39] but I agree that we need a more clear weekly description on what needs to be updated in the train... the ready to deploy column doesn't seem informative enough [10:05:00] joal: by that you mean watch them in hue as they progress right? [10:05:57] I think it is also watching that whatever changed led to the desired effects [10:06:25] in this case it seems webrequest, but I don't have context on the change [10:07:19] fdans: I mean checking hue, and checking that page-title looks ok in newly-computed hours [10:08:25] we could have a permanent etherpad called /analytics-train with dates and few lines each time [10:08:33] about what to do/check [10:08:52] and possibly how (not always straightforward) [10:09:20] it is probably a bit more painful in the beginning but eventually we'll all level up [10:09:36] (not to Joseph level that is impossible but hopefully something close :P) [10:10:08] thoughts? [10:11:29] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Update wikimedia-history revision data with deleted field (and find it a new name?) - https://phabricator.wikimedia.org/T178587 (10JAllemandou) Data is available in new test-datasource located at /user/joal/wmf/data/wmf/mediawiki/user_history: ` spark.rea... [10:18:36] 10Analytics, 10Analytics-Kanban: Add caused_by_user_text to mediawiki_page_history - https://phabricator.wikimedia.org/T167608 (10JAllemandou) ` spark.read.parquet("/user/joal/wmf/data/wmf/mediawiki/page_history/snapshot=2019-03").createOrReplaceTempView("mwph") spark.sql("select caused_by_user_text, count(1)... [10:21:39] "Yes Luca it sounds awesome! Why don't you send an email to the team so anybody can tell you their ideas?" [10:21:54] Makes sense I'll do it, thanks for the feedback [10:21:57] :D [10:24:18] awwww elukey sorry, I tend to space out from irc when I'm with wikistats stuff [10:24:54] Please excuse me elukey - coding-friday, talking away [10:27:22] yeah all excuses! :D [10:31:33] * elukey lunch! [11:15:58] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Add user_is_bot_by to MediaWiki history - https://phabricator.wikimedia.org/T219177 (10JAllemandou) ` spark.read.parquet("/user/joal/wmf/data/wmf/mediawiki/history/snapshot=2019-03").createOrReplaceTempView("mwh") spark.sql("selec... [11:19:47] 10Analytics, 10Analytics-Kanban, 10Anti-Harassment, 10Product-Analytics: Add partial blocks to mediawiki history tables - https://phabricator.wikimedia.org/T211950 (10JAllemandou) a:03JAllemandou [12:03:53] 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, 10Product-Analytics: mediawiki_history missing page events - https://phabricator.wikimedia.org/T205594 (10JAllemandou) Checking improvements in new datsource. ` // Current datasource - normally the problem is present in here spark.read.parquet("/wm... [13:07:00] o/ elukey: any more thoughts on https://gerrit.wikimedia.org/r/c/operations/puppet/+/504787 ? [13:07:00] i'm sure it will need more testing/patches, but to the general layout and ideaa? [13:13:42] (03CR) 10Nuria: Fix mediawiki-history user event join (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/504834 (owner: 10Joal) [13:13:46] (03CR) 10Nuria: [C: 04-1] Fix mediawiki-history user event join [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/504834 (owner: 10Joal) [13:17:32] Good morning all! Just a reminder that your cloud VMs are in danger of breaking for good due to pending DNS changes: https://phabricator.wikimedia.org/T221408 [13:19:41] 10Analytics: Puppet broken on most VMs in the 'analytics' project - https://phabricator.wikimedia.org/T221408 (10Ottomata) a:03elukey Most of the VMs here are for Hadoop testing, pinging @elukey to check it out. :) [13:21:41] ottomata: o/ [13:21:53] thanks andrewbogott! [13:22:11] ottomata: I have some follow up questions but only for curiosity [13:22:33] 1) will the schema registry be accessible by external clients? [13:22:46] if so, what kind of traffic abuse protection is needed? [13:23:01] (03CR) 10Nuria: Update mw user-history timestamps (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/497604 (https://phabricator.wikimedia.org/T218463) (owner: 10Joal) [13:23:05] 2) what happens if the schema registry is not available? (say a ton of traffic hits it and it is unable to answer [13:23:23] probably I am missing some bits so apologies in advance :) [13:24:01] elukey: for most things schemasa will be checked out locally anyway in the container imagae [13:24:09] and they are also cached [13:24:24] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: AQS alerts due to big queries issued to Druid for the edit API - https://phabricator.wikimedia.org/T219910 (10Nuria) 05Open→03Resolved [13:24:29] (answering 2 first) [13:24:37] so eventgate only asks for the schemas the first time it needs them [13:24:42] and if they are not found locally [13:24:54] prod critical stuff will not be configured to use the schema registrys [13:24:56] only analytics stuff [13:24:57] ah yes now I remember, when the pod is created? [13:25:04] ack ack [13:25:04] no, when the image is built by CI [13:25:09] ahhhh [13:25:13] so you have to aactually trigger it by pushing aa taga [13:25:18] pushing a tag* [13:25:21] but, mostly [13:25:27] i want this service up ssooner rather than later not for eventgate [13:25:28] but for refine [13:25:38] so changes are not dynamic (namely they are not picked up on the fly) [13:25:49] otherwise I have to puppetize something to deploy aand update schemas repos in hdfs [13:26:01] elukey: for the pods, no [13:26:11] its baked into the docker image [13:26:22] the schema service will have the latest stuff available [13:26:27] ok answering 1: [13:26:43] wait first one more on 2: [13:26:53] also, this is no worse than meta.wm.org schemas now [13:27:01] if that goes down, all of eventlogging breaks [13:27:11] its a little better since some schemas will be local [13:27:18] only new recent changes will need to be looked up [13:27:57] ok 1: [13:28:03] i don't plan on making it public yet [13:28:19] we'll want it to be public so that analysts can browse schemas, etc. [13:28:27] but i dont' actually plan on making schema.wm.org yet... [13:28:32] maybe I should just remove that from the server_name for now [13:28:41] ah okok [13:28:42] and just use schema.svc.etc [13:29:07] when public [13:29:12] i don't think we'll need much traffic abuse protection [13:29:15] its all static files [13:29:19] cacheable by varnish [13:29:35] yep makes sense, I was only reasoning out loud [13:29:49] don't have more comments, the rest seems fine for me [13:30:00] ok! i'm going to remove the schema.wm.org part until we need it [13:30:03] otherwise it might be confusing [13:30:57] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Core Platform Team Backlog (Watching / External), and 2 others: Modern Event Platform: Stream Intake Service: Implementation - https://phabricator.wikimedia.org/T206785 (10Nuria) [13:30:58] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review: EventGate Helm chart should POST test event for readinessProbe - https://phabricator.wikimedia.org/T218680 (10Nuria) 05Open→03Resolved [13:31:30] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Ingest data from PrefUpdate EventLogging schema into Druid - https://phabricator.wikimedia.org/T218964 (10Nuria) 05Open→03Resolved [13:31:44] ottomata: I am seeing the value of having it public though, people will easily check schemas etc.. without any ssh tunnel (but probably only needed for later) [13:31:51] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Upgrade matomo1001 to latest upstream - https://phabricator.wikimedia.org/T218037 (10Nuria) 05Open→03Resolved [13:32:17] i mean, the schemas are also in git and therefore github [13:32:19] so also browsable [13:32:32] 10Analytics, 10Analytics-EventLogging, 10QuickSurveys, 10Readers-Web-Backlog (Tracking): QuickSurveys EventLogging missing ~10% of interactions - https://phabricator.wikimedia.org/T220627 (10Nuria) p:05Unbreak!→03Triage [13:32:39] but we will see, the analystis reaally didn't want to give up nicely browsesable and discoverable schemas [13:32:45] the UI on this might get nicer in the future [13:33:15] 10Analytics, 10Analytics-EventLogging, 10QuickSurveys, 10Readers-Web-Backlog (Tracking): QuickSurveys EventLogging missing ~10% of interactions - https://phabricator.wikimedia.org/T220627 (10Nuria) Moving to radar as further steps of code chnages to Quicksurveys to fix loading issues with JS should be done... [13:33:35] ok elukey thanks, gimme a +1 or something. walking home real quick, then will do some PCCs and a merge and get this thing going [13:33:36] 10Analytics, 10Analytics-Kanban, 10Wikimedia-Incident: attempt to backfill eventlogging data from eventlogging-client-side topic into per schema topics - https://phabricator.wikimedia.org/T220421 (10Nuria) p:05Triage→03High [13:33:46] once it is in place I can more easily do the schema based refine for eventbus data! [13:34:27] 10Analytics, 10Dumps-Generation: pageviews dumps contain invalid lines - https://phabricator.wikimedia.org/T217071 (10Nuria) [13:34:37] 10Analytics, 10Analytics-Kanban, 10Datasets-General-or-Unknown, 10Patch-For-Review, and 2 others: Pageview dumps incorrectly formatted, need to escape special characters - https://phabricator.wikimedia.org/T144100 (10Nuria) [13:37:21] 10Analytics, 10Product-Analytics: Eventbus revisions are duplicated in event.mediawiki_revision_tags_change - https://phabricator.wikimedia.org/T218246 (10Nuria) Moving to radar and ping to @Pchelolo that tags changed events look to be sent twice. [13:38:28] 10Analytics, 10Analytics-Kanban, 10MediaWiki-Vagrant, 10Patch-For-Review: Vagrant initial provision fails on NodeJS version mismatch - https://phabricator.wikimedia.org/T218238 (10Nuria) 05Open→03Resolved [13:38:45] 10Analytics, 10Analytics-Kanban, 10Core Platform Team (Modern Event Platform (TEC2)), 10Core Platform Team Backlog (Watching / External), and 2 others: [Post-mortem] Kafka Jumbo cluster cannot accept connections - https://phabricator.wikimedia.org/T219842 (10Nuria) 05Open→03Resolved [13:40:15] danke [13:47:11] oh elukey i meant to ask you [13:47:13] about aanalaytics labs [13:47:24] did you mean to apply kerberos to all nodes in that project? [13:47:33] i tried to spawn a new node but puppet failed because of that [13:49:09] (03CR) 10Nuria: Remove leftover files in oozie folders (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/504914 (owner: 10Joal) [13:51:29] ottomata: ah no no I think it was done by Moritz IIRC when we were testing [13:51:34] feel free to remove the settings [13:51:42] I think that we can also delete the hadoop instances for the moment [13:51:49] they are not really needed [13:51:53] I can do it now if you want [13:54:24] whenever you like i'm not messing with it now [14:05:57] joal: qq - do you know how tbayer_popups was created in druid? The product analytics folks are not using it, so I think we could drop it, but in case anybody will need it again (i doubt but still) I'd like to know if it will be possible to re-create it [14:10:55] 10Analytics, 10Operations, 10Research-management, 10Patch-For-Review, 10User-Elukey: Remove computational bottlenecks in stats machine via adding a GPU that can be used to train ML models - https://phabricator.wikimedia.org/T148843 (10elukey) Had an interesting chat with Gilles today about his use case.... [14:18:17] brb [14:56:16] 10Analytics, 10Operations, 10Research-management, 10Patch-For-Review, 10User-Elukey: Remove computational bottlenecks in stats machine via adding a GPU that can be used to train ML models - https://phabricator.wikimedia.org/T148843 (10Nuria) >so the plan would be to install thumbor on stat1005 How would... [14:59:28] 10Analytics, 10Operations, 10Research-management, 10Patch-For-Review, 10User-Elukey: Remove computational bottlenecks in stats machine via adding a GPU that can be used to train ML models - https://phabricator.wikimedia.org/T148843 (10elukey) I think the idea would be to run it with/without GPU active an... [15:06:17] ottomata: was the extra eventbus lvs config for 8190 meant to be added to the schema.svc patch? [15:06:29] (asking to be sure, I reviewed it now) [15:06:40] eventbus lvs? [15:06:53] https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/505254/4/conftool-data/service/services.yaml [15:07:05] shoot no thanks. sorry am too hasty [15:08:29] I am a bit rusty in LVS config so I don't recall what that part configs, but it seemed odd given the rest :D [15:09:09] elukey: https://gerrit.wikimedia.org/r/c/operations/puppet/+/505258 [15:11:03] super [15:15:04] (03CR) 10Elukey: "> All good on the ones already done, some miss (property to be added," [analytics/refinery] - 10https://gerrit.wikimedia.org/r/504846 (https://phabricator.wikimedia.org/T220971) (owner: 10Elukey) [15:15:30] (03PS2) 10Elukey: Swap hdfs user with analytics [analytics/refinery] - 10https://gerrit.wikimedia.org/r/504846 (https://phabricator.wikimedia.org/T220971) [15:17:42] SOME CHANGESET elukey https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/504846/! [15:19:39] ahahahah [15:19:51] the worst part will be the deploy :P [15:24:34] elukey: IIRC tbayer_popups was created when testing virtual_pageviews - I''m gonna double check with him :) [15:25:00] yep yep I remember that [15:25:05] joal: no, that was another schema [15:25:20] joal: we can delete tbayer_popups w/o worries [15:25:50] joal: popups is the prequel to teh previews but events were in two different schemas [15:26:18] ok nuria - Thanks for the good memory :) [15:26:44] * joal needs to buy more cold-storage [15:28:59] I already asked to product analytics, they don't use it [15:29:54] 10Analytics, 10Analytics-Kanban: Remove dead code from refinery/oozie folders - https://phabricator.wikimedia.org/T221460 (10JAllemandou) [15:30:26] 10Analytics, 10Analytics-Kanban: Remove dead code from refinery/oozie folders - https://phabricator.wikimedia.org/T221460 (10JAllemandou) a:03JAllemandou [15:30:59] (03PS2) 10Joal: Remove leftover files in oozie folders [analytics/refinery] - 10https://gerrit.wikimedia.org/r/504914 (https://phabricator.wikimedia.org/T221460) [15:31:01] elukey: NUKE it ! [15:31:42] (03CR) 10Joal: Remove leftover files in oozie folders (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/504914 (https://phabricator.wikimedia.org/T221460) (owner: 10Joal) [15:33:14] 10Analytics, 10Analytics-Kanban: Decide: start_timestamp for mediawiki history - https://phabricator.wikimedia.org/T220507 (10JAllemandou) The `user` part of this task is in testing with the datasource located at `hdfs:///user/joal/wmf/data/wmf/mediawiki/history/snaphsot=2019-03` and `hdfs:///user/joal/wmf/dat... [15:45:08] (03CR) 10Joal: [C: 03+1] "Not merging because it is to be synchronized with other patches, but ready IMO" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/504846 (https://phabricator.wikimedia.org/T220971) (owner: 10Elukey) [15:49:06] (03CR) 10Ottomata: "Should we do this a little more slowly? Like one or a few jobs at a time?" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/504846 (https://phabricator.wikimedia.org/T220971) (owner: 10Elukey) [15:58:51] (03CR) 10Nuria: [C: 03+2] Remove leftover files in oozie folders [analytics/refinery] - 10https://gerrit.wikimedia.org/r/504914 (https://phabricator.wikimedia.org/T221460) (owner: 10Joal) [16:01:08] ping ottomata standduppp [16:01:28] (03CR) 10Elukey: "> Should we do this a little more slowly? Like one or a few jobs at" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/504846 (https://phabricator.wikimedia.org/T220971) (owner: 10Elukey) [16:02:22] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Core Platform Team (Modern Event Platform (TEC2)), and 3 others: Schema Registry HTTP Service - https://phabricator.wikimedia.org/T219552 (10Ottomata) [16:18:27] 10Analytics, 10EventBus, 10Core Platform Team (Modern Event Platform (TEC2)), 10Core Platform Team Backlog (Next), 10Services (next): Factor lib/kafka.js out of eventgate and change-propagation into its own library - https://phabricator.wikimedia.org/T220725 (10Ottomata) a:05Ottomata→03None [16:33:19] elukey: you coming to this kafka main meeting? [16:33:22] no worries if not! [16:33:26] yep yep! [16:33:29] I was talking with Fran [16:42:54] Hey a-team! The HelpPanel whitelist patch was merged last week, and if I remember correctly that would then have been deployed this past Wednesday. I don't see a "helppanel" table in the event_sanitized database, though. Could someone look into that? [16:54:43] I have another question about software on the stat machines. Is there any way to enable x-forwarding over ssh? [16:55:20] my preferred tool for the kind of work i'm doing right now is x-forwarded emacs :) [16:59:44] hm, that I do not know, perhapss via an ssh tunnel you could do that? [17:01:09] googling seems to tell me it is not enabled for the ssh server by default [17:01:41] i thiink fdans does some kind of remote editor thing. probabaly not x11 but some kind of auto fs sync? [17:04:02] groceryheist: ottomata I use Transmit, my favourite mac app ever [17:05:50] 10Analytics, 10Operations, 10Research-management, 10Patch-For-Review, 10User-Elukey: Remove computational bottlenecks in stats machine via adding a GPU that can be used to train ML models - https://phabricator.wikimedia.org/T148843 (10dr0ptp4kt) (Detour) @Nuria the other day I mentioned my project aroun... [17:08:16] fdans, joal : i think we can fix the state of master with the latest known good gerrit commit , let me create a branch with it [17:08:39] Nettrom: we have not deployed this week, we deploy most wednesdays but not all [17:10:15] nuria: okay, will this be deployed next week then? [17:11:58] Nettrom: yes it will [17:12:11] joal: this is the last good patch on master right? https://gerrit.wikimedia.org/r/#/c/analytics/refinery/source/+/504382/ [17:12:43] nuria: Thanks! I'll check in on this again next week. [17:13:14] * elukey off! [17:13:21] have a good weekend people o/ [17:13:33] Nettrom: yes, in general we deploy most wed but it could be that 1) we run into issues 2) too many people are on vacation .. etc [17:13:48] ottomata: I have enabled ForwardX11 and ForwardX11Trusted in my .ssh/config [17:20:00] 10Analytics: trying to get a clean master branch - https://phabricator.wikimedia.org/T221466 (10Nuria) [17:20:27] nuria: Of course, that's perfectly reasonable! One challenge for me is that there's little information to find about this process. Last I looked, the whitelisting documentation doesn't set expectations on timeframes or explain that part of the process. There's also not been any communication about timeframe on the phab task. Now that I know all this, I don't need it, of course. [17:22:09] nuria: nonono, do you want to bc and I'll update you on findings with joseph? [17:22:32] fdans: sure [17:22:35] omw [17:29:57] Nettrom: ah ya, sorry, this is pretty new procedure, chnages to whitelist took place imediately as of recent [17:31:17] * changes that is, will correct doc [17:49:08] nuria: array items? :) [17:49:44] ottomata: give me 15 mins? [17:50:00] sure! [18:00:14] (03PS1) 10Nuria: Trying to get a clean master branch [analytics/refinery/source] (master-2019-04) - 10https://gerrit.wikimedia.org/r/505271 (https://phabricator.wikimedia.org/T221466) [18:00:58] joal, fdans : i created a new branch on gerrit with our last good change on master: https://gerrit.wikimedia.org/r/#/q/status:open+project:analytics/refinery/source+branch:master-2019-04 [18:01:56] joal, fdans : this should have the right history (let's check) and i think it can be used to obliterate master's poor state [18:02:10] joal, fdans : or, in the worst case we move from master to here [18:02:20] ottomata: yt? [18:02:33] YA [18:02:38] ottomata: bc [18:02:42] ? [18:12:55] (03PS2) 10Nuria: Getting a clean master branch [analytics/refinery/source] (master-2019-04) - 10https://gerrit.wikimedia.org/r/505271 (https://phabricator.wikimedia.org/T221466) [18:12:59] (03CR) 10Nuria: [C: 03+2] Getting a clean master branch [analytics/refinery/source] (master-2019-04) - 10https://gerrit.wikimedia.org/r/505271 (https://phabricator.wikimedia.org/T221466) (owner: 10Nuria) [18:13:12] (03CR) 10Nuria: [V: 03+2 C: 03+2] Getting a clean master branch [analytics/refinery/source] (master-2019-04) - 10https://gerrit.wikimedia.org/r/505271 (https://phabricator.wikimedia.org/T221466) (owner: 10Nuria) [18:14:22] fdans, joal: and this is the new master branch on github that should have the clean history: https://github.com/wikimedia/analytics-refinery-source/blob/master-2019-04/changelog.md [18:21:48] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Discovery, and 4 others: Rewrite Avro schemas (ApiAction, CirrusSearchRequestSet) as JSONSchema and produce to EventGate - https://phabricator.wikimedia.org/T214080 (10Ottomata) [18:58:25] 10Analytics, 10EventBus, 10Operations, 10Core Platform Team (Modern Event Platform (TEC2)), and 3 others: Possibly expand Kafka main-{eqiad,codfw} clusters in Q4 2019. - https://phabricator.wikimedia.org/T217359 (10herron) Today we discussed desired hardware configs and expansion strategies during a meetin... [19:18:30] Nettrom: Please see: https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging/Data_retention_and_auto-purging#When_are_whitelist_changes_effective [19:26:24] 10Analytics, 10MediaWiki-extensions-GrowthExperiments, 10Product-Analytics, 10Growth-Team (Current Sprint): Homepage: instrumentation - https://phabricator.wikimedia.org/T216586 (10JTannerWMF) [19:30:25] nuria: awesome, thanks for adding it! [19:38:18] joal, fdans : i think easiest would be to delete master brnach and replace it with the one i created but deleting master branch on gerrit is not doable [19:45:49] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Discovery, and 4 others: Rewrite Avro schemas (ApiAction, CirrusSearchRequestSet) as JSONSchema and produce to EventGate - https://phabricator.wikimedia.org/T214080 (10Ottomata) Yahoo we have cirrussearch-request events in beta! [20:16:20] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Discovery, and 4 others: Rewrite Avro schemas (ApiAction, CirrusSearchRequestSet) as JSONSchema and produce to EventGate - https://phabricator.wikimedia.org/T214080 (10Nuria) Nice! [21:18:04] 10Analytics, 10Product-Analytics, 10Epic, 10User-Elukey: Provide feature parity between the wiki replicas and the Analytics Data Lake - https://phabricator.wikimedia.org/T212172 (10Neil_P._Quinn_WMF) [22:02:18] 10Analytics, 10Product-Analytics: Identify imported revisions in `mediawiki_history` - https://phabricator.wikimedia.org/T221482 (10Neil_P._Quinn_WMF) [22:20:23] 10Analytics, 10Product-Analytics: Identify imported revisions in mediawiki_history - https://phabricator.wikimedia.org/T221482 (10Neil_P._Quinn_WMF) [22:44:38] 10Analytics, 10Analytics-Cluster, 10Operations: furud - DISK CRITICAL - /mnt/hdfs is not accessible: Input/output error - https://phabricator.wikimedia.org/T221483 (10Dzahn) [22:47:24] 10Analytics, 10Analytics-Cluster, 10Operations: furud - DISK CRITICAL - /mnt/hdfs is not accessible: Input/output error - https://phabricator.wikimedia.org/T221483 (10Dzahn) 05Open→03Resolved a:03Dzahn followed the docs at https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administrat... [23:48:39] 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10WMF-JobQueue, and 2 others: Beta cluster: MassMessage fails with PHP fatal error because of Declaration of JobQueueEventBus::doAck() must be compatible with that of JobQueue::doAck() - https://phabricator.wikimedia.org/T220662 (10DannyS712) @kostajh Accordin...