[02:31:50] 10Analytics, 10Gerrit, 10Gerrit-Privilege-Requests, 10Patch-For-Review, 10User-MarcoAurelio: Give access to Wikistats 2 to l10n-bot - https://phabricator.wikimedia.org/T245805 (10abi_) Thanks @MarcoAurelio, that seems like the likely cause. I've deployed the patch, exports will be run tomorrow. Will upda... [03:51:24] 10Analytics, 10Analytics-Kanban: Vet high volume bot spike detection code - https://phabricator.wikimedia.org/T238363 (10Nuria) a:03JAllemandou [03:56:54] 10Analytics, 10Analytics-Kanban, 10Better Use Of Data, 10Desktop Improvements, and 7 others: Enable client side error logging in prod for small wiki - https://phabricator.wikimedia.org/T246030 (10Nuria) @Tgr This is mediaWiki newbies 101 .. how can i see that wikimedia events extension is deployed to test... [04:42:36] 10Analytics, 10Event-Platform, 10Product-Analytics, 10CPT Initiatives (Modern Event Platform (TEC2)): Eventbus revisions are duplicated in event.mediawiki_revision_tags_change - https://phabricator.wikimedia.org/T218246 (10Nuria) Seems like any fix here requires digging into mediawiki code to see where tho... [05:01:02] (03CR) 10Milimetric: [C: 03+1] "this looks great, not sure what order we want to merge stuff" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/576278 (https://phabricator.wikimedia.org/T246745) (owner: 10Fdans) [05:19:38] 10Analytics, 10Analytics-Kanban, 10Better Use Of Data, 10Desktop Improvements, and 7 others: Enable client side error logging in prod for small wiki - https://phabricator.wikimedia.org/T246030 (10Milimetric) Deployed extensions can be seen on any wiki at /Special:Version, which on hawaiian wikipedia is at... [05:21:00] 10Analytics, 10Gerrit, 10Gerrit-Privilege-Requests, 10User-MarcoAurelio: Give access to Wikistats 2 to l10n-bot - https://phabricator.wikimedia.org/T245805 (10Milimetric) Thank you so much yall! [05:39:57] 10Analytics: Create table in hive with a continent lookup for countries - https://phabricator.wikimedia.org/T127995 (10Milimetric) Hm... isn't this task literally making two very simple files? One would be a list of country codes with the continent they belong to, and another would be a create table statement.... [06:16:29] 10Analytics, 10Analytics-Kanban, 10Better Use Of Data, 10Desktop Improvements, and 7 others: Enable client side error logging in prod for small wiki - https://phabricator.wikimedia.org/T246030 (10Tgr) An easier although probably less educational way to interpret the page is by looking at https://haw.wikipe... [06:28:27] 10Analytics, 10Analytics-Kanban, 10Better Use Of Data, 10Desktop Improvements, and 7 others: Enable client side error logging in prod for small wiki - https://phabricator.wikimedia.org/T246030 (10Tgr) Error reporting is disabled for that wiki: `lang=javascript > mw.config.get( 'wgWMEClientErrorIntakeURL' )... [07:10:59] (03PS5) 10Fdans: Add the ability to manually select locales to build [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/576278 (https://phabricator.wikimedia.org/T246745) [08:14:48] (03PS1) 10Lex Nasser: Create intermediate geoeditors table for loading into Cassandra [analytics/refinery] - 10https://gerrit.wikimedia.org/r/576618 (https://phabricator.wikimedia.org/T244597) [08:14:59] (03CR) 10Fdans: "@Milimetric order here doesn't really matter, the patches don't conflict with each other" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/576278 (https://phabricator.wikimedia.org/T246745) (owner: 10Fdans) [08:38:22] !log Kill-restart mediawiki-history-dumps-coord [08:38:23] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:41:17] !log Kill-restart mediawiki-history-reduced-coord [08:41:22] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:46:31] bonjour :) [08:46:48] Hi elukey - just taking a minute to restart jobs, then back to kid's day :) [08:47:12] ttl! [08:48:13] elukey: bonjour :) [08:48:35] just tried again the rsync [08:51:02] djellel: hi! I am waiting for a puppet change to merge, SRE is doing maintenance, I hope that it fixes it [08:51:06] ~10 mins ETA [09:10:05] a little bit more sorry [09:16:11] djellel: it should work now [09:17:34] elukey: perfect, it works, thank you [09:18:59] gooood [10:07:11] joal: Thanks again for the review :-D, I was able to pull data that looks sane, and the Spark shell has been really nice for random exploration. In case you're curious, here's a raw doc with the explorations: https://etherpad.wikimedia.org/p/conflict_queries [11:33:02] * elukey lunch! [12:36:30] nice awight :) [12:36:48] awight: a trick I think you have not used: caching dataframes :) [13:43:29] joal: That sounds excellent. I'll read about caching, I'm not sure yet which datasets would be broadcast. I guess the big win would be the initial "conflicts" set before joining against the larger revision table... [13:44:07] other than that, datasets are all tiny (5k rows) so I imagine it makes no difference whether we cache for the last few joins. [13:44:25] awight: I was more thinking to cache the conflicts_clean one, in order to make exploration easy (small-ish data) [13:45:08] awight: main join is done once, so no real advantage of caching - however caching for exploration is nice [13:53:05] aah I'll try that right now [13:55:09] 10Analytics, 10Core Platform Team, 10Event-Platform, 10Product-Analytics, 10CPT Initiatives (Modern Event Platform (TEC2)): Eventbus revisions are duplicated in event.mediawiki_revision_tags_change - https://phabricator.wikimedia.org/T218246 (10WDoranWMF) [14:19:07] 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10serviceops: Create production and canary releases for existent eventgate helmfile services - https://phabricator.wikimedia.org/T245203 (10akosiaris) [14:19:43] 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10serviceops: Create production and canary releases for existent eventgate helmfile services - https://phabricator.wikimedia.org/T245203 (10akosiaris) I 've merged https://gerrit.wikimedia.org/r/576402 and manually cleaned the various IPVS services that we... [14:21:38] oh much improved, thanks for the optimization! [14:28:25] 10Analytics, 10Operations, 10User-Elukey: Refactor Analytics POSIX groups in puppet to improve maintainability - https://phabricator.wikimedia.org/T246578 (10elukey) I tried to add a script that periodically enforces proper home dir permissions, but I had to revert since the admin module is of course already... [14:48:00] 10Analytics, 10Analytics-Kanban, 10Better Use Of Data, 10Desktop Improvements, and 7 others: Enable client side error logging in prod for small wiki - https://phabricator.wikimedia.org/T246030 (10Ottomata) > Error reporting is disabled for that wiki: haw yes, we were testing in on test.wikipedia.org first.... [14:51:52] 10Analytics, 10Core Platform Team, 10Event-Platform, 10Product-Analytics, 10CPT Initiatives (Modern Event Platform (TEC2)): Eventbus revisions are duplicated in event.mediawiki_revision_tags_change - https://phabricator.wikimedia.org/T218246 (10Ottomata) Hm yeah if the events have different `meta.id`s, t... [14:52:20] 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10serviceops: Create production and canary releases for existent eventgate helmfile services - https://phabricator.wikimedia.org/T245203 (10Ottomata) YES THANK YOU! [15:24:21] ottomata: o/ [15:30:14] (03CR) 10Milimetric: [C: 03+2] Add the ability to manually select locales to build [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/576278 (https://phabricator.wikimedia.org/T246745) (owner: 10Fdans) [15:31:24] (03Merged) 10jenkins-bot: Add the ability to manually select locales to build [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/576278 (https://phabricator.wikimedia.org/T246745) (owner: 10Fdans) [15:33:26] (03CR) 10Milimetric: [C: 03+2] Fix i18n key mismatch - month over month (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/576282 (https://phabricator.wikimedia.org/T246750) (owner: 10Fdans) [15:34:12] (03CR) 10Milimetric: [C: 03+2] Prevent sidebar from widening too much [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/576289 (https://phabricator.wikimedia.org/T246744) (owner: 10Fdans) [15:34:29] thanks for the reviews milimetric ! [15:35:43] (03Merged) 10jenkins-bot: Fix i18n key mismatch - month over month [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/576282 (https://phabricator.wikimedia.org/T246750) (owner: 10Fdans) [15:35:45] (03Merged) 10jenkins-bot: Prevent sidebar from widening too much [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/576289 (https://phabricator.wikimedia.org/T246744) (owner: 10Fdans) [15:41:44] elukey: hello! [15:41:51] ottomata: goooood morning [15:42:26] I'd need to bring my parent's cat to the vet in about half an hour, it is ok if we move the ops sync for another time/day? [15:44:58] ottomata: --^ [15:47:25] ya sure! [15:47:39] we can just skip this week elukey i feel synced :) [15:47:52] ack thanks! [15:48:46] ottomata: when you have a minute if you want to test random stuff in Hadoop test I'd be super happy [15:48:56] just to see what I have missed [15:48:59] in test cluster? [15:49:08] correct [15:49:24] k [15:49:36] in theory everything should work, in practice I am sure I forgot stuff [15:49:40] :D [16:12:33] 10Analytics, 10Wikidata, 10Wikidata-Campsite (Wikidata-Campsite-Iteration-∞): Add time limits to scripts executed on stat1007 as part of analytics/wmde/scripts - https://phabricator.wikimedia.org/T243894 (10Addshore) 05Open→03Resolved [16:12:36] 10Analytics, 10Wikidata, 10Wikidata-Campsite: Long query running on dbstore1005:3318 - https://phabricator.wikimedia.org/T243871 (10Addshore) [16:13:03] 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10serviceops, 10Patch-For-Review: Create production and canary releases for existent eventgate helmfile services - https://phabricator.wikimedia.org/T245203 (10Ottomata) [16:16:22] 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10serviceops, 10Patch-For-Review: Create production and canary releases for existent eventgate helmfile services - https://phabricator.wikimedia.org/T245203 (10Ottomata) [16:16:38] elukey: https://gerrit.wikimedia.org/r/c/operations/homer/public/+/576873 [16:21:11] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Better Use Of Data, and 9 others: Modern Event Platform: Stream Configuration: Implementation - https://phabricator.wikimedia.org/T233634 (10Ottomata) [16:21:16] 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10serviceops, 10Patch-For-Review: Create production and canary releases for existent eventgate helmfile services - https://phabricator.wikimedia.org/T245203 (10Ottomata) [16:22:35] 10Analytics, 10Analytics-Kanban, 10serviceops: Clarify multi-service instance concepts in helm charts and enable canary releases - https://phabricator.wikimedia.org/T242861 (10Ottomata) @akosiaris for my purposes I'm satisfied, but I'm not sure we settled this totally for other charts. Most controversial wa... [16:26:21] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 6 others: Public EventGate instance and endpoint for analytics event intake: eventgate-analytics-external - https://phabricator.wikimedia.org/T233629 (10Ottomata) [16:32:07] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 6 others: Public EventGate instance and endpoint for analytics event intake: eventgate-analytics-external - https://phabricator.wikimedia.org/T233629 (10Ottomata) [16:38:28] (03PS3) 10Mforns: Add dimensions to druid pageview_hourly and virtualpageview_hourly [analytics/refinery] - 10https://gerrit.wikimedia.org/r/570681 (https://phabricator.wikimedia.org/T243090) [16:40:46] 10Analytics, 10Analytics-Kanban, 10Better Use Of Data, 10Desktop Improvements, and 8 others: Enable client side error logging in prod for small wiki - https://phabricator.wikimedia.org/T246030 (10LGoto) [16:45:11] 10Analytics: Wiki selector: "All wikis" is not translated - https://phabricator.wikimedia.org/T246911 (10fdans) [16:47:03] 10Analytics: Metric widget won't stretch when content overflows - https://phabricator.wikimedia.org/T246913 (10fdans) [16:49:09] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Infrastructure-Team-Backlog: Explore sending batches of events from EPC libraries - https://phabricator.wikimedia.org/T239996 (10jlinehan) a:05jlinehan→03mpopov [16:55:09] 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10serviceops, 10Patch-For-Review: Create production and canary releases for existent eventgate helmfile services - https://phabricator.wikimedia.org/T245203 (10Ottomata) [16:56:39] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 6 others: Public EventGate instance and endpoint for analytics event intake: eventgate-analytics-external - https://phabricator.wikimedia.org/T233629 (10Ottomata) [17:18:25] elukey: https://gerrit.wikimedia.org/r/c/operations/homer/public/+/576873 :D [17:18:47] 10Analytics, 10Product-Analytics (Kanban): Update wmfdata to support multiple SQL engines for Hive databases - https://phabricator.wikimedia.org/T246060 (10nshahquinn-wmf) a:05nshahquinn-wmf→03mpopov The big pull request is complete: https://github.com/neilpquinn/wmfdata/pull/8! Assigning to Mikhail sinc... [17:18:52] ottomata: ah nice! I can merge it later if you want [17:18:57] yes plz ty [17:18:59] whenever [17:20:01] (03PS1) 10Fdans: Release 2.7.1 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/576904 [17:20:19] 10Analytics, 10Epic, 10Product-Analytics (Kanban): Analysts cannot reliably use wmfdata to run SQL queries against Hive databases - https://phabricator.wikimedia.org/T245891 (10nshahquinn-wmf) [17:22:27] (03CR) 10Fdans: [V: 03+2 C: 03+2] Release 2.7.1 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/576904 (owner: 10Fdans) [17:39:30] Gone for diner, back in a bit [17:55:45] be back in a bit! [18:53:22] * elukey off! [19:12:36] PROBLEM - Check if the Hadoop HDFS Fuse mountpoint is readable on notebook1003 is CRITICAL: connect to address 10.64.21.109 port 5666: Connection refused https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration%23Fixing_HDFS_mount_at_/mnt/hdfs [19:19:57] 10Analytics, 10Cloud-Services, 10Developer-Advocacy, 10Documentation: Set up a "WMCS Edits Dashboard" page on Meta - https://phabricator.wikimedia.org/T246932 (10srishakatux) [19:20:04] 10Analytics, 10Cloud-Services, 10Developer-Advocacy, 10Documentation: Set up a "WMCS Edits Dashboard" page on Meta - https://phabricator.wikimedia.org/T246932 (10srishakatux) p:05Triage→03High [19:40:03] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Add new dimensions to virtual_pageview_hourly and pageview_hourly - https://phabricator.wikimedia.org/T243090 (10mforns) @cchen Hi! I finished the code for this task and I'm testing the loading of the corresponding data, and I see t... [19:53:07] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Add new dimensions to virtual_pageview_hourly and pageview_hourly - https://phabricator.wikimedia.org/T243090 (10cchen) @mforns thanks for working on this! We only need project_family and namespace data for pageviews table. [19:59:49] (03PS4) 10Mforns: Add dimensions to druid pageview_hourly and virtualpageview_hourly [analytics/refinery] - 10https://gerrit.wikimedia.org/r/570681 (https://phabricator.wikimedia.org/T243090) [20:06:24] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Add new dimensions to virtual_pageview_hourly and pageview_hourly - https://phabricator.wikimedia.org/T243090 (10mforns) Thanks @cchen! @Nuria was there another reason to also populate virtualpageview_hourly? [20:37:37] heya mforns , qq, got any ideas for how to make a window.onerror event fire in thhe browser? [20:41:34] hey ottomata [20:42:23] ottomata, can you explain? [20:43:00] so, we are trying to get mw client side to auto log errors to a backend [20:43:15] aha [20:43:17] mw.errorLogger registers a window.onerror handler to do this [20:43:25] ok [20:43:33] i can manually do what it does in the console, i.e. mw.track('global.error', ...) [20:43:34] and that works [20:43:43] but i want to get a real error via the window.onerror handler [20:44:31] ottomata, is mw.errorLogger's handler defining a callback like in: https://blog.sentry.io/2016/01/04/client-javascript-reporting-window-onerror ? [20:44:38] yes [20:45:09] if so, it'd get the full info about the error, including lineNo, colNo, and error obj [20:45:11] https://github.com/wikimedia/mediawiki/blob/master/resources/src/mediawiki.base/mediawiki.errorLogger.js#L142-L186 [20:45:16] k [20:47:05] ottomata, I see, but then, what is the problem? isn't errorLogger doing what you want? [20:47:16] mforns: i've not gotten an event via window.onerror [20:47:33] i can manually type mw.track('global.error', ...) into the console [20:47:34] and it works [20:47:37] but i want a real error [20:47:43] not one I've manually constructed [20:49:00] AHHII GOT ONE [20:49:05] just from clicking around! [20:49:31] oooh! I get that xD [20:49:58] maybe try to execute a function that is undefined? [20:50:09] i tried that [20:50:10] object.nonexistingfield(); [20:50:13] oh [20:51:09] yeh i t seems the console is catching these kinds of errors [20:56:59] 10Analytics, 10Analytics-Kanban, 10Better Use Of Data, 10Desktop Improvements, and 8 others: Enable client side error logging in prod for small wiki - https://phabricator.wikimedia.org/T246030 (10Ottomata) Ok, things are finally working on test.wikipedia.org! I have events! :) Will enable in Hawaiian wi... [21:23:24] hey a-team, I'm hearing that notebook1003 is down, which is affecting the analysts - any word on what happened and when it'll be back up? [21:55:43] looking [21:59:43] kzeta: it looks like someone used up all the memory a couple of hours ago, but it looks back to normal now [22:00:15] can you access? [22:01:11] relaying to my team; there was some discussion in our Slack channel and Connie mentioned it in our 1:1 [22:02:20] ottomata: thanks for checking! [22:03:10] kzeta: I ssh-ed to the box and I see a bunch of processes running for Nuria and Amir so it looks like the box is ok, but I did find something [22:03:17] kzeta: it looks like the HDFS Fuse mount broke [22:03:42] so maybe while they were trying to get files off of HDFS, if they're doing that through /mnt/hdfs, that would've broken [22:04:27] milimetric: l is it broken now? [22:04:43] ottomata: the nagios alert shows CRITICAL and hasn't recovered, lemme check tho [22:04:47] ok [22:05:14] it's not even there in /mnt [22:05:19] so... I'm guessing that's bad [22:05:39] do de do [22:05:40] https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration#Fixing_HDFS_mount_at_%2Fmnt%2Fhdfs [22:05:41] but get this, it doesn't say No such file or directory, it says: [22:05:42] -bash: cd: hdfs: Input/output error [22:06:07] cided [22:06:10] fixed* [22:06:34] (I don't have sudo on that box and I think 'cause I copy/pasted now someone got an illegal sudo attempt warning and they're coming after me) [22:06:38] ya i fixeed [22:06:40] (therefore I'm gonna go hide) [22:06:43] haha [23:16:15] thanks! Connie confirmed that it's working for her [23:54:51] 10Analytics, 10Cloud-Services, 10Developer-Advocacy, 10Documentation: Set up a "WMCS Edits Dashboard" page on Meta - https://phabricator.wikimedia.org/T246932 (10srishakatux) Published here: https://meta.wikimedia.org/wiki/WMCS_Edits_Dashboard [23:57:53] 10Analytics, 10Cloud-Services, 10Developer-Advocacy (Jan-Mar 2020): Further improvements to the WMCS edits dashboard - https://phabricator.wikimedia.org/T240040 (10srishakatux) [23:57:56] 10Analytics, 10Cloud-Services, 10Developer-Advocacy (Jan-Mar 2020): Modify ReportUpdater to support `YYYY-MM` dates for monthly reports - https://phabricator.wikimedia.org/T245096 (10srishakatux) [23:58:17] 10Analytics: Modify ReportUpdater to support `YYYY-MM` dates for monthly reports - https://phabricator.wikimedia.org/T245096 (10srishakatux)