[06:59:10] 10Analytics, 10Analytics-Kanban: LDAP ldap-ro.eqiad.wikimedia.org not reachable from Analytics VLAN - https://phabricator.wikimedia.org/T227611 (10elukey) The services in analytics that use LDAP are: * Hue * Jupyter notebooks * Yarn (via httpd) * Superset (via httpd) * Turnilo (via httpd) The last three will... [06:59:15] 10Analytics, 10Analytics-Kanban: LDAP ldap-ro.eqiad.wikimedia.org not reachable from Analytics VLAN - https://phabricator.wikimedia.org/T227611 (10elukey) [10:16:47] 10Analytics, 10Tool-Pageviews: Load media requests data into cassandra (or druid?) - https://phabricator.wikimedia.org/T228149 (10fdans) [10:50:30] 10Analytics, 10Tool-Pageviews: The mediacounts dataset doesn't have a project dimension - https://phabricator.wikimedia.org/T228151 (10fdans) [10:50:55] 10Analytics, 10Tool-Pageviews: Load media requests data into cassandra (or druid?) - https://phabricator.wikimedia.org/T228149 (10fdans) [10:51:14] 10Analytics, 10Tool-Pageviews: Load media requests data into cassandra (or druid?) - https://phabricator.wikimedia.org/T228149 (10fdans) [11:14:31] (03PS10) 10Fdans: Add file extension and media type classification to media files UDF [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/517641 (https://phabricator.wikimedia.org/T225911) [11:15:22] (03CR) 10Fdans: "nuria: i added another test to make sure that types are consolidated to their long-form, e.g. jpg's file_type should be stored as jpeg." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/517641 (https://phabricator.wikimedia.org/T225911) (owner: 10Fdans) [11:16:22] * elukey lunch! [11:17:31] (03CR) 10jerkins-bot: [V: 04-1] Add file extension and media type classification to media files UDF [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/517641 (https://phabricator.wikimedia.org/T225911) (owner: 10Fdans) [11:19:58] 10Analytics, 10Tool-Pageviews: Load media requests data into cassandra (or druid?) - https://phabricator.wikimedia.org/T228149 (10fdans) [11:20:02] 10Analytics-Kanban, 10Patch-For-Review: Add access-site and access-type to mediacounts job - https://phabricator.wikimedia.org/T225910 (10fdans) [11:21:11] 10Analytics-Kanban, 10Patch-For-Review: Add access-site and access-type to mediacounts job - https://phabricator.wikimedia.org/T225910 (10fdans) As discussed, since access site data doesn't include mobile web requests, it doesn't add value to include this field in the dataset, so we'll only be including agent... [11:22:05] 10Analytics, 10Tool-Pageviews: Load media requests data into cassandra (or druid?) - https://phabricator.wikimedia.org/T228149 (10fdans) [11:22:12] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Expand regex that maps file types to media - https://phabricator.wikimedia.org/T225911 (10fdans) [11:22:56] 10Analytics, 10Services (watching): Add mediacounts data to AQS and, from there, Restbase - https://phabricator.wikimedia.org/T207208 (10fdans) [11:23:00] 10Analytics, 10Tool-Pageviews: Load media requests data into cassandra (or druid?) - https://phabricator.wikimedia.org/T228149 (10fdans) [11:23:17] 10Analytics, 10Tool-Pageviews: Load media requests data into cassandra (or druid?) - https://phabricator.wikimedia.org/T228149 (10fdans) [11:23:23] 10Analytics, 10Tool-Pageviews: Statistics for views of individual Wikimedia images - https://phabricator.wikimedia.org/T210313 (10fdans) [11:23:44] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Expand regex that maps file types to media - https://phabricator.wikimedia.org/T225911 (10fdans) [11:23:47] 10Analytics, 10Services (watching): Add mediacounts data to AQS and, from there, Restbase - https://phabricator.wikimedia.org/T207208 (10fdans) [11:27:42] 10Analytics-Kanban, 10Patch-For-Review: Add and backfill agent-type in mediacounts hive dataset - https://phabricator.wikimedia.org/T225910 (10fdans) [12:15:51] 10Analytics: User knissen can't access Superset - https://phabricator.wikimedia.org/T226431 (10kai.nissen) This seems stalled, is there anything I can do to help? [13:08:03] 10Analytics: User knissen can't access Superset - https://phabricator.wikimedia.org/T226431 (10elukey) Checked on the db and the username seems ok: ` MariaDB [superset_production]> select * from ab_user where username = 'knissen'; +-----+------------+-----------+----------+----------+--------+------------------... [13:20:36] (03PS3) 10Fdans: Add access type to mediacounts hourly dataset [analytics/refinery] - 10https://gerrit.wikimedia.org/r/517426 (https://phabricator.wikimedia.org/T225910) [13:24:37] 10Analytics, 10Tool-Pageviews: The mediacounts dataset doesn't have a project dimension - https://phabricator.wikimedia.org/T228151 (10Tgr) It should not be hard to parse the referer and find out which project the browser is referred from. Knowing where the image was uploaded is probably good for something but... [13:28:50] 10Analytics: User knissen can't access Superset - https://phabricator.wikimedia.org/T226431 (10elukey) Tried to delete your user and dashboard to restart from a clean state, but apparently I was only able to do the latter (your user still holds some data that I don't know where it is defined). Let's try again wi... [13:29:14] elukey: i just saw your update in relation to kniss.en [13:30:09] and wanted to point out a specific bit in the error messages, specificly `"Duplicate entry '-' for key 'email'")` [13:30:58] so it seems something is trying to do an insert with the email filed set to '-' however that value has allready been used and [im gussing] it a unique field so the insert is being rejected [13:31:00] jbond42: hey! yes the date matches with the creation of the username.. Nuria mentioned in the task that a piece of config was missing, so I am almost sure that the email was not there upon first insertion [13:31:43] ok cool, i dont know anything about this app but just wanted to make sure you spattig that bit [13:31:55] s/spattig/spotted/ [13:32:03] thanks a lot! [13:32:06] np [13:36:18] 10Analytics, 10Cloud-Services, 10observability, 10Patch-For-Review, and 2 others: High Prometheus TCP retransmits - https://phabricator.wikimedia.org/T225296 (10elukey) https://phabricator.wikimedia.org/T153468 is relevant for this task, since the last patch needs also to allow IPv6 addresses for the prome... [13:40:04] 10Analytics, 10Operations, 10Traffic, 10User-Elukey: TLS certificates for Analytics origin servers - https://phabricator.wikimedia.org/T227860 (10elukey) The idea that I have is to re-use what done for the appservers, namely put nginx in front of httpd to terminate TLS. In theory we could: * generate one... [13:41:25] 10Analytics, 10Operations, 10Traffic, 10User-Elukey: TLS certificates for Analytics origin servers - https://phabricator.wikimedia.org/T227860 (10elukey) [13:41:49] 10Analytics, 10Analytics-Kanban, 10Operations, 10Traffic, 10User-Elukey: TLS certificates for Analytics origin servers - https://phabricator.wikimedia.org/T227860 (10elukey) a:03elukey [13:43:50] elukey: i'm here [13:45:47] i could login to superset and am still asked to re-authenticate when trying to view a dashboard. [13:48:20] Kai_WMDE: hey! [13:48:35] 10Analytics: User knissen can't access Superset - https://phabricator.wikimedia.org/T226431 (10kai.nissen) Not sure, if this is a problem, but the email address is wrong (`knissen@wikimedia.de` instead of `kai.nissen@wikimedia.de`). [13:51:35] 10Analytics: User knissen can't access Superset - https://phabricator.wikimedia.org/T226431 (10elukey) >>! In T226431#5337013, @kai.nissen wrote: > Not sure, if this is a problem, but the email address is wrong (`knissen@wikimedia.de` instead of `kai.nissen@wikimedia.de`). Fixed! In theory it shouldn't be an is... [13:54:43] so Kai_WMDE, to recap - you can see the list of dashboards, but with any of them you get a 401 [13:55:08] elukey: I am asked to re-authenticate when trying to view one [13:55:28] Kai_WMDE: and if you authenticate, do you see the dashboard? [13:55:32] if I provide my credentials again, the dashboard loads [13:55:42] but this is happening on every request [13:56:20] 10Analytics, 10EventBus: [WIP] RFC: Stream Configuration Service - https://phabricator.wikimedia.org/T227906 (10Ottomata) 05Open→03Declined Thanks so much for the ResourceLoader idea, Timo! It already does pretty much everything we'd need. I've talked with @jlinehan, and he thinks that keeping these conf... [13:56:24] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Core Platform Team (Modern Event Platform (TEC2)), and 3 others: Modern Event Platform: Stream Configuration Service - https://phabricator.wikimedia.org/T205319 (10Ottomata) [13:56:41] all right now it is clearer, when you wrote in the task the first time I thought it was something different (401 every time without loading) [13:57:48] 10Analytics: Jan Dittrich would like to have access to superset - https://phabricator.wikimedia.org/T227093 (10elukey) @Jan_Dittrich does it work now? [13:59:50] Kai_WMDE: then we circle back to https://phabricator.wikimedia.org/T224159 [14:00:07] but IIRC you told me that you tried different browsers right? [14:00:25] Can you list in the task your os and browser/versions tested ? [14:00:44] 10Analytics, 10Analytics-Kanban, 10Operations, 10Traffic, 10User-Elukey: TLS certificates for Analytics origin servers - https://phabricator.wikimedia.org/T227860 (10Ottomata) Hm, all for it! Although, do you think it would be worth exploring the built in TLS support in the services where they support i... [14:01:32] 10Analytics: Jan Dittrich would like to have access to superset - https://phabricator.wikimedia.org/T227093 (10Jan_Dittrich) it does, thanks! [14:01:40] 10Analytics, 10Analytics-Kanban, 10Operations, 10Traffic, 10User-Elukey: TLS certificates for Analytics origin servers - https://phabricator.wikimedia.org/T227860 (10elukey) >>! In T227860#5337057, @Ottomata wrote: > Hm, all for it! Although, do you think it would be worth exploring the built in TLS sup... [14:01:57] 10Analytics: Jan Dittrich would like to have access to superset - https://phabricator.wikimedia.org/T227093 (10elukey) 05Open→03Resolved [14:02:53] 10Analytics: User knissen can't access Superset - https://phabricator.wikimedia.org/T226431 (10elukey) After a chat with Kai on IRC, it seems that this is a re-occurrence of https://phabricator.wikimedia.org/T224159 [14:03:51] seems to work on firefox now [14:04:22] only an error message when loading a dashboard. [14:04:24] "There was an issue fetching the favorite status of this dashboard." [14:05:59] 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10User-Elukey: Import AMD rocm packages in wikimedia-buster - https://phabricator.wikimedia.org/T224723 (10elukey) Refactored the puppet code into a separate module called amd_rocm and updated the documentation. We'll need to follow up wi... [14:08:48] Kai_WMDE: there are probably some issue with the current version of superset and some browsers, I hope that it will get fixed for the next release.. [14:18:54] elukey: I see. Seems that only adding favorites is not working, which is an expendable feature. :) [14:19:02] thanks for helping [14:20:04] elukey: is it documented how an event logging data source can be added? [14:21:41] Kai_WMDE: yes [14:21:41] https://wikitech.wikimedia.org/wiki/Analytics/Systems/Hive_to_Druid_Ingestion_Pipeline [14:21:42] but [14:21:56] it is mostly manual now; we'd have to set it up for you [14:22:11] we hope to automate this process much better over the next few quarters [14:23:25] ottomata: as FYI https://phabricator.wikimedia.org/T226778 [14:23:29] commented on all the tasks [14:23:41] next week it will be interesting [14:23:41] :D [14:24:03] nice ok [14:38:42] 10Analytics, 10Operations, 10hardware-requests, 10User-Elukey: eqiad: 1 misc node for the Kerberos KDC service - https://phabricator.wikimedia.org/T227288 (10elukey) @wiki_willy @RobH hi! Don't mean to jump the queue, but I am wondering if this task and its codfw one could be prioritized over the next week... [14:41:09] (03PS3) 10Fdans: Add file extension and media classification to mediacounts job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/522390 (https://phabricator.wikimedia.org/T225911) [14:46:39] (03PS1) 10Fdans: Delete limn-flow-data queries in favor of reportupdater-queries [analytics/limn-flow-data] - 10https://gerrit.wikimedia.org/r/523731 (https://phabricator.wikimedia.org/T222739) [14:49:03] (03PS1) 10Fdans: Delete limn-edit-data queries in favor of reportupdater-queries [analytics/limn-edit-data] - 10https://gerrit.wikimedia.org/r/523732 (https://phabricator.wikimedia.org/T222739) [14:50:25] hello people, I am going to stop eventlogging mysql consumers to allow a reboot of db1107 [14:50:28] as FYI [14:51:33] (03PS1) 10Fdans: Delete limn-language-data queries in favor of reportupdater-queries [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/523733 (https://phabricator.wikimedia.org/T222739) [14:51:42] elukey: ok! [14:51:47] 10Analytics, 10Operations, 10ops-eqiad: Broken disk on analytics1072 - https://phabricator.wikimedia.org/T226467 (10Cmjohnson) I received the disk on-site but I cannot tell which disk is failed, they all have green LEDs. @elukey could you please let me know which disk slot or let's coordinate to make the di... [14:53:23] (03PS1) 10Fdans: Delete limn-ee-data queries in favor of reportupdater-queries [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/523734 (https://phabricator.wikimedia.org/T222739) [14:56:17] (03CR) 10Fdans: [C: 03+1] pageview: move the oozie coordinator to hive2 actions [analytics/refinery] - 10https://gerrit.wikimedia.org/r/523200 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [14:57:18] 10Analytics, 10Operations, 10hardware-requests, 10User-Elukey: eqiad: 1 misc node for the Kerberos KDC service - https://phabricator.wikimedia.org/T227288 (10wiki_willy) @elukey @RobH - I've marked it as accelerate on the procurement doc. Rob, can you work on getting these two servers included on this pro... [15:01:33] 10Analytics, 10Analytics-EventLogging, 10DBA, 10Operations, 10ops-eqiad: db1107 (eventlogging db master) possibly memory issues - https://phabricator.wikimedia.org/T222050 (10Cmjohnson) one last paste of the idrac log Record: 84 Date/Time: 04/29/2019 09:35:48 Source: system Severity: Non... [15:06:47] 10Analytics, 10Analytics-EventLogging, 10DBA, 10Operations, 10ops-eqiad: db1107 (eventlogging db master) possibly memory issues - https://phabricator.wikimedia.org/T222050 (10Cmjohnson) Swapped DIMM A3 with DIMM B3, now we have to powrer the server back on and let it go for a few days to see if the error... [15:10:47] fdans: browser reports seem to have stopped in May, did you see that as part of moving jobs around? [15:12:05] June 6th [15:12:06] hm [15:18:57] milimetric: o/ anything that I'd need to check? [15:19:14] seems like something that I may have broken with a puppet change or something :D [15:19:40] elukey: mmm, not sure, it's that same old problem of .pid files left around after probably a restart or something like that [15:20:15] I removed the .pid files manually, jobs should run [15:20:37] but I remember we declined my fancy ideas for how to prevent the problem in the future [15:24:06] milimetric: but did the ru job failed silently? [15:24:11] yea [15:24:27] there's an error in the log, but we never got any email from the timer [15:24:38] systemctl status reportupdater-browser.timer [15:25:20] systemctl status reportupdater-browser.service right? [15:25:55] if I do journalctl -u reportupdater-browser I see the errors [15:26:11] (this is on stat7 btw, I'm sure you know) [15:26:42] just seen the errors, ru probably doesn't exit 1 when this happens [15:26:45] is it possible? [15:27:13] checking [15:27:31] 10Analytics, 10EventBus, 10Operations, 10Core Platform Team (Modern Event Platform (TEC2)), and 3 others: Possibly expand Kafka main-{eqiad,codfw} clusters in Q4 2019. - https://phabricator.wikimedia.org/T217359 (10RobH) [15:28:37] indeed, you're right elukey [15:35:47] (03PS1) 10Milimetric: Fail when exiting improperly [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/523747 [15:38:19] elukey: but the timer seems to have stopped logging a few weeks ago [15:39:10] uh... maybe that's just me not understanding journalctl [15:41:04] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Fail when exiting improperly [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/523747 (owner: 10Milimetric) [15:41:17] (03CR) 10Elukey: Fail when exiting improperly (031 comment) [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/523747 (owner: 10Milimetric) [15:41:25] left a nit [15:41:29] ah, sorry [15:41:30] nothing stopping :) [15:41:40] no no it was about logging.exception, might have been handier [15:41:41] no blocker [15:42:06] ah, yeah, all the string formatting and logging needs to be updated repo-wide. I just had to fix that one line because it was failing linting [15:42:13] +! [15:42:14] +1 [15:42:37] but yea, figured I should merge this just in case a whole bunch of other jobs are failing and we don't know [15:44:59] milimetric: about the logging, I can see [15:45:00] Jul 16 15:00:01 stat1007 reportupdater-browser[117058]: 2019-07-16 15:00:01,338 - ERROR - Could not open or parse the pid file [15:45:03] Jul 16 15:00:01 stat1007 reportupdater-browser[117058]: 2019-07-16 15:00:01,338 - WARNING - Another instance is already running. Exiting. [15:45:06] that seems recent no? [15:45:20] yeah, I did -u without -f [15:45:24] my bad [15:45:25] ahhh okok [15:45:30] ack :) [15:45:53] should run again in a few minutes, and succeed because the pid is gone [15:46:14] and then an hour later maybe we'll get an alert if it's still catching up and tries to run again [15:46:26] which would be a good test that the sys.exit works [15:48:14] 10Analytics, 10Operations, 10ops-eqiad: Broken disk on analytics1072 - https://phabricator.wikimedia.org/T226467 (10elukey) Seems to have worked: ` elukey@analytics1072:~$ sudo megacli -PDList -aALL | grep "Firmware state" Firmware state: Unconfigured(good), Spun Up Firmware state: Online, Spun Up Firmware... [16:02:48] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 4 others: Make it possible to use $ref in JSONSchemas - https://phabricator.wikimedia.org/T206824 (10Ottomata) [16:03:08] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 4 others: Make it possible to use $ref in JSONSchemas - https://phabricator.wikimedia.org/T206824 (10Ottomata) a:03Ottomata [16:06:09] nuria: standup? [16:20:21] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Core Platform Team Backlog (Watching / External), and 3 others: Modern Event Platform: Stream Intake Service: Migrate Mediawiki Eventbus events to eventgate-main - https://phabricator.wikimedia.org/T211248 (10Ottomata) a:05Pchelolo→03Ottomata [16:23:47] (03CR) 10Milimetric: [C: 03+1] "all tags closed correctly /me very afraid I missed something else but that's how oozie is" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/523200 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [16:28:17] (03CR) 10Elukey: "> all tags closed correctly /me very afraid I missed something else" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/523200 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [16:29:24] 10Analytics, 10Analytics-Dashiki: Pie charts not showing on "User Agent Breakdowns" dashboard - https://phabricator.wikimedia.org/T228187 (10Esanders) [16:29:34] 10Analytics, 10Analytics-Dashiki: Pie charts not showing on "User Agent Breakdowns" dashboard - https://phabricator.wikimedia.org/T228187 (10Esanders) p:05Triage→03High [16:31:17] 10Analytics, 10Analytics-Dashiki, 10Analytics-Kanban: Pie charts not showing on "User Agent Breakdowns" dashboard - https://phabricator.wikimedia.org/T228187 (10Milimetric) a:03Milimetric @Esanders: we just saw that this morning, reports were stuck, fixed now, running to catch up, should be fine in an hour... [16:38:48] * elukey off! [16:43:18] 10Analytics: Page creation data stream died June 6 - https://phabricator.wikimedia.org/T228188 (10kaldari) [16:58:59] 10Analytics: Page creation data stream died June 6 - https://phabricator.wikimedia.org/T228188 (10kaldari) The previous breakage task for reference: T201420 [17:16:56] AndyRussG: o/ [17:17:35] ottomata: here! just one sec, someone's t the door ;p [17:18:09] k np i'm still setting up [17:21:37] k AndyRussG ready when you are [17:24:25] ottomata: back! [17:24:32] sorry about that! [17:24:45] I was just getting an URL ready for u to test with [17:24:45] ok! [17:24:47] one sec [17:25:04] ok first lets test that the events work as expected, you'll just need to make a change to an existing campaign i guess? [17:27:20] ottomata: no there is a way to force the event [17:27:33] however... not working just now? hmmm [17:28:20] AndyRussG: there's no way for you to trigger the eevent? [17:28:31] even in e.g. test.wikipedia.org or something? [17:28:51] oh soryr, you are saying there IS a way [17:28:55] misread [17:28:56] ok [17:29:21] it's ok [17:29:42] ottomata: this should be working, but actually it's not triggering the event! https://en.wikinews.org/wiki/Main_Page?uselang=bn&randomcampaign=0.1&force=true&impressionEventSampleRate=1 [17:30:30] ya don't see it either [17:31:34] ottomata: actually it must be something is broken on the client-side [17:31:35] AndyRussG: i do see that some change events were emitted a couple of hours ago [17:31:47] ottomata: oh wait sorry [17:31:56] right those are the events we're checking [17:32:04] OK, yes I do need to change something in a campaign! [17:32:09] well, create is fine [17:32:10] or delete [17:32:12] Aaaargh where is my brain! [17:32:20] K yes I'll make a change in a campaign [17:32:24] we should check all three even, but checking one is probably enough for confidence :) [17:32:26] ok [17:32:52] delete is just there for show, it's never actually triggered currently [17:34:12] ottomata: I just made a change [17:34:27] so you should see an event for that [17:35:25] I do!~ [17:35:26] great. [17:35:26] ok [17:35:45] AndyRussG: is it possible for you to make the change trigger using mwdebug1002? [17:35:50] via X-Wikimedia-Debug header? [17:37:12] ottomata: sure [17:37:19] one sec [17:37:21] great, go ahead, i've scap pulled the config change there [17:38:22] great! looks perfect [17:38:29] i'm going to scap sync to fleet [17:39:11] ottomata: ok great! [17:39:19] AndyRussG: is it easy to trigger a create event? [17:42:41] AndyRussG: ^ [17:42:41] ? [17:42:43] ottomata: sure [17:42:46] do u want one? [17:42:50] yup go ahead [17:42:52] okok [17:43:44] doing so on the debug host [17:44:08] k, i've deployed everywhere so you can do on any host [17:44:54] got it! [17:44:58] great stuff, thanks AndyRussG [17:45:02] all good. [17:46:06] ottomata: cool beans! [17:46:17] thanks so much for this eh! [17:47:11] yeah! [17:47:16] AndyRussG: should we be importing these events into hive? [17:47:18] that would be useful, yes? [17:47:26] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Core Platform Team Backlog (Watching / External), and 3 others: Modern Event Platform: Stream Intake Service: Migrate Mediawiki Eventbus events to eventgate-main - https://phabricator.wikimedia.org/T211248 (10Ottomata) [17:47:35] ottomata: isn't that where they're going? [17:47:52] there's part of a new pipeline that we're just finally getting back to finishing [17:47:52] as far as I see they aren't configured to. [17:48:01] Ah yes that's where we'd like 'em pls [17:48:01] they could be very easily tho! [17:48:03] maybe we never just set that up [17:48:04] ok [17:48:15] ottomata: aaaargh [17:48:29] sorry my brain is... no I was thinking of the other kind of event again [17:48:33] apologies [17:48:37] the EVentLogging ones? [17:48:43] ottomata: no, these events don't need to go to hive [17:48:44] hes [17:48:46] yes [17:48:48] ok [17:48:48] dsljffdsalkjdsafñlkjdsañlkjfdsañlkjfdsa [17:48:51] :) [17:48:54] they could very easily if you want them [17:49:00] should I really drink more coffee???!! [17:49:14] ottomata: nah, let's not put 'em in hive for now [17:49:19] k [17:49:21] sounds like extra resources that aren't really needed [17:49:36] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Core Platform Team Backlog (Watching / External), and 3 others: Modern Event Platform: Stream Intake Service: Migrate Mediawiki Eventbus events to eventgate-main - https://phabricator.wikimedia.org/T211248 (10Ottomata) [17:49:37] I'll let you know if at some point it's useful though, thanks! [17:49:44] sure [18:11:15] 10Analytics, 10Operations, 10hardware-requests, 10User-Elukey: eqiad: 1 misc node for the Kerberos KDC service - https://phabricator.wikimedia.org/T227288 (10RobH) [18:11:22] 10Analytics, 10Operations, 10hardware-requests, 10User-Elukey: codfw: 1 misc node for the Kerberos KDC service - https://phabricator.wikimedia.org/T227425 (10RobH) [18:12:33] 10Analytics, 10Operations, 10hardware-requests, 10User-Elukey: eqiad: 1 misc node for the Kerberos KDC service - https://phabricator.wikimedia.org/T227288 (10RobH) a:03elukey Please note there are two requests currently open to add a single cpu misc host for Kerberos KDC and Kadmin daemons, one in codfw... [18:12:46] 10Analytics, 10Operations, 10hardware-requests, 10User-Elukey: codfw: 1 misc node for the Kerberos KDC service - https://phabricator.wikimedia.org/T227425 (10RobH) a:03elukey Please note there are two requests currently open to add a single cpu misc host for Kerberos KDC and Kadmin daemons, one in codfw... [18:27:46] 10Analytics, 10Operations, 10hardware-requests, 10User-Elukey: codfw: 1 misc node for the Kerberos KDC service - https://phabricator.wikimedia.org/T227425 (10elukey) Asking confirmation to @MoritzMuehlenhoff since these hosts might be used by SRE too, but from my point of view the difference is ok (and the... [18:45:56] 10Analytics, 10Operations, 10SRE-Access-Requests: Requesting access to stats machines/ores hosts hosts for Andy Craze - https://phabricator.wikimedia.org/T226204 (10Halfak) Looks like we forgot 'deployment' and 'deploy-services' groups. I've filed {T228191} to add those too. [18:53:12] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Core Platform Team Backlog (Watching / External), and 3 others: Modern Event Platform: Stream Intake Service: Migrate Mediawiki Eventbus events to eventgate-main - https://phabricator.wikimedia.org/T211248 (10Ottomata) [19:19:34] 10Analytics, 10Core Platform Team, 10EventBus, 10Services (later): revision-create events are sometimes emitted in a secondary DC - https://phabricator.wikimedia.org/T207994 (10Pchelolo) Moving tis to icebox since it contains some good info, but we are unlikely to work on this [19:24:10] 10Analytics, 10EventBus, 10Core Platform Team (Code Health (TEC13)): Factor lib/kafka.js out of eventgate and change-propagation into its own library - https://phabricator.wikimedia.org/T220725 (10Pchelolo) p:05Normal→03Low [19:30:47] 10Analytics, 10Core Platform Team, 10Wikimedia-Stream, 10Documentation: stream.wikimedia.org/?doc returns an error page - https://phabricator.wikimedia.org/T227958 (10Pchelolo) [19:59:29] 10Analytics, 10Analytics-EventLogging, 10DBA, 10Operations, 10ops-eqiad: db1107 (eventlogging db master) possibly memory issues - https://phabricator.wikimedia.org/T222050 (10Cmjohnson) 05Open→03Resolved Resolving this task for now, if the error returns please re-open and ping me. [20:20:32] 10Analytics, 10Operations, 10ops-eqiad: Broken disk on analytics1072 - https://phabricator.wikimedia.org/T226467 (10Cmjohnson) @elukey the disk has been replaced, it is in still unconfigured (good) the disk needs to be mapped back to Virtual Drive: 1 (Target Id: 1) Slot Number: 0 This is a little out of my... [20:22:02] 10Analytics, 10Operations, 10ops-eqiad: Broken disk on analytics1072 - https://phabricator.wikimedia.org/T226467 (10Cmjohnson) cmjohnson@analytics1072:~$ sudo megacli -LdPdInfo -aall | grep -e 'Virtual Drive' -e Slot Virtual Drive: 0 (Target Id: 0) Slot Number: 12 Slot Number: 13 Virtual Drive: 2 (Target Id... [21:14:02] 10Analytics, 10Discovery-Analysis, 10Product-Analytics, 10Patch-For-Review: Productionize per-country daily & monthly active app user stats - https://phabricator.wikimedia.org/T186828 (10kzimmerman) Chelsy had an oozie job updating 2 tables daily/monthly: mobile_apps-uniques-by_country-daily-coord and mobi... [21:30:08] 10Analytics, 10EventBus, 10Growth-Team, 10Notifications, 10Wikimedia-production-error: Database error "Duplicate entry" for PRIMARY key (from EchoNotificationMapper::insert) - https://phabricator.wikimedia.org/T217079 (10kostajh) The spacing on this is kind of weird ([query](https://logstash.wikimedia.or...