[07:00:04] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Shortcut icon is not showing - https://phabricator.wikimedia.org/T197482#4301562 (10sahil505) [07:00:07] 10Analytics-Kanban, 10Google-Summer-of-Code (2018): [Analytics] Improvements to Wikistats2 front-end - https://phabricator.wikimedia.org/T189210#4301561 (10sahil505) [07:02:25] 10Analytics, 10Operations, 10ops-eqiad, 10User-Elukey: Degraded RAID on dbstore1002 - https://phabricator.wikimedia.org/T197707#4301564 (10elukey) p:05Triage>03High a:03elukey [07:03:37] 10Analytics, 10Operations, 10ops-eqiad, 10User-Elukey: Degraded RAID on dbstore1002 - https://phabricator.wikimedia.org/T197707#4300251 (10elukey) @RobH, @Cmjohnson - is it possible to swap the disk even if warranty is expired? We are not ready yet to decom this host (but will anticipate its hw replacement... [07:23:57] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Google-Summer-of-Code (2018): GSoC Proposal 2018 : [Analytics] Improvements to Wikistats2 front-end - https://phabricator.wikimedia.org/T189964#4301590 (10sahil505) [07:27:51] 10Analytics, 10Operations, 10SRE-Access-Requests: Requesting access for mbsantos - https://phabricator.wikimedia.org/T197237#4301599 (10Joe) We also need @greg approval for adding people to deployers. [07:31:16] 10Analytics, 10Operations, 10ops-eqiad, 10User-Elukey: Degraded RAID on dbstore1002 - https://phabricator.wikimedia.org/T197707#4301603 (10Marostegui) These are 1TB disks. ``` Raw Size: 1.090 TB [0x8bba0cb0 Sectors] ``` [08:06:59] 10Analytics, 10Analytics-Kanban, 10EventBus, 10ORES, and 4 others: Modify revision-score schema so that model probabilities won't conflict - https://phabricator.wikimedia.org/T197000#4301699 (10awight) I'm not sure if it's in a format that might be helpful for this task, but we have a JSON-schema defined f... [08:22:35] 10Analytics, 10Analytics-Kanban, 10EventBus, 10ORES, and 4 others: Modify revision-score schema so that model probabilities won't conflict - https://phabricator.wikimedia.org/T197000#4275600 (10Gaboe420) El El mar, 12 de junio de 2018 a la(s) 07:38, Ottomata < no-reply@phabricator.wikimedia.org> escribió:... [08:42:37] joal: o/ - this seems interesting http://bigtop.apache.org/ [08:44:06] (hoping that https://wiki.debian.org/Hadoop is not true anymore) [11:47:10] I am trying to think if bigtop could be an alternative to CDH [11:47:36] not in the immediate future but maybe long term it would be great to have some sort of upstream contact/support [12:34:10] 10Analytics, 10Analytics-Kanban: Archive old geowiki data (editors per country) and make it easily available at WMF - https://phabricator.wikimedia.org/T190856#4302563 (10fdans) Re-sqooping of `geowiki_archive_active_editors_world` done :) [13:18:01] 10Analytics, 10Operations, 10ops-eqiad, 10User-Elukey: Degraded RAID on dbstore1002 - https://phabricator.wikimedia.org/T197707#4302652 (10Cmjohnson) These servers are out of warranty, I do not think I have any 1TB disks in the data center but I can use a 2TB. [13:28:11] o/ I'm around [13:28:21] feeling better today!!!! [13:41:31] heyaaaaa [13:47:29] 10Analytics, 10Analytics-Kanban: Archive old geowiki data (editors per country) and make it easily available at WMF - https://phabricator.wikimedia.org/T190856#4302719 (10mforns) `geowiki_archive_active_editors_world` looks good to me now! [14:51:25] 10Analytics, 10Analytics-Kanban: Archive old geowiki data (editors per country) and make it easily available at WMF - https://phabricator.wikimedia.org/T190856#4302935 (10mforns) `geowiki_archive_monthly_country` Looks good to me! There's only one small detail that we can discuss whether we want to change or n... [15:03:56] 10Analytics, 10Analytics-Kanban: Archive old geowiki data (editors per country) and make it easily available at WMF - https://phabricator.wikimedia.org/T190856#4303045 (10mforns) `geowiki_archive_monthly_edits_country` Looks good to me overall as well. There's another small difference in relation to `geowiki_a... [15:17:41] 10Analytics, 10Analytics-Kanban: Archive old geowiki data (editors per country) and make it easily available at WMF - https://phabricator.wikimedia.org/T190856#4303108 (10mforns) The `geowiki_archive_monthly` data in Turnilo looks good to me overall and super useful! There is an empty metric called `Count` tho... [15:36:42] 10Analytics, 10Analytics-Kanban: Archive old geowiki data (editors per country) and make it easily available at WMF - https://phabricator.wikimedia.org/T190856#4303158 (10mforns) In Superset the 'Geowiki legacy archive' dashboard works well and shows correct data. I had difficulties seeing the countries though... [15:49:29] Hi nuria_ [15:58:28] is anyone available to help with an eventlogging problem? we're not seeing events get into the database on deployment-eventlog05 and we're not sure how to debug. nuria_ said to reach out on IRC [15:59:33] Hi bearloga - We're moving inot standup time now, but we'll have time just after [16:02:31] joal: cool, thanks! :) [16:03:02] bearloga: tried to restart eventlogging, some processors listed some kafka failures.. can you re check? [16:05:01] elukey: checking… [16:33:42] Ok team, I confirm the exact scenario I talked about with Superset [16:34:34] elukey joal: seeing events in client-side-events.log but not all-events.log. everything seems alright [16:35:07] Great bearloga [16:35:12] https://www.irccloud.com/pastebin/DbPEe0kw/ [16:35:22] I can't say why you don't see them in all-events.log though [16:35:52] https://meta.wikimedia.org/wiki/Schema:MobileWikiAppLanguageSearching is the schema [16:36:05] is there a reason why that event wouldn't get validated? [16:36:29] bearloga: non valid events appear on the error logs, let me send you doc one sec [16:37:41] bearloga: doc here, let me check is up to date as our migration to latest linux changed location of logs: https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging/TestingOnBetaCluster#Validated_events [16:39:50] which one to check? `eventlogging-processor@client-side-00.log` or `eventlogging-processor@client-side-01.log`? [16:42:19] bearloga: is this event blacklisted for mySQL? [16:43:13] nope [16:46:09] nuria_: at least, it shouldn't be. We didn't ask for events from that schema to be blocked from going into mysql [16:47:06] 10Analytics, 10Analytics-EventLogging, 10Readers-Web-Backlog: Some VirtualPageView are too long and fail EventLogging processing - https://phabricator.wikimedia.org/T196904#4303412 (10Jdlrobson) [16:49:09] (03PS1) 10Thcipriani: Scap: Remove git_server from scap.cfg [analytics/pivot/deploy] - 10https://gerrit.wikimedia.org/r/441236 (https://phabricator.wikimedia.org/T162814) [16:50:58] (03PS1) 10Thcipriani: Scap: Remove git_server from scap.cfg [analytics/aqs/deploy] - 10https://gerrit.wikimedia.org/r/441237 (https://phabricator.wikimedia.org/T162814) [16:52:27] (03PS1) 10Thcipriani: Scap: Remove git_server from scap.cfg [analytics/refinery/scap] - 10https://gerrit.wikimedia.org/r/441238 (https://phabricator.wikimedia.org/T162814) [16:53:38] (03PS1) 10Thcipriani: Scap: Remove git_server from scap.cfg [analytics/turnilo/deploy] - 10https://gerrit.wikimedia.org/r/441239 (https://phabricator.wikimedia.org/T162814) [16:56:02] bearloga: i only see one validation error: [16:56:04] bearloga: [16:56:05] 16:44:57 deployment-eventlog05 eventlogging-processor@client-side-00[13675]: 2018-06-20 16:44:57,190 [13675] (MainThread) Unable to validate: [16:56:05] ?%7B%22schema%22%3A%22MobileWikiAppLanguageSearching%22%2C%22revision%22%3A18110978%2C%22wiki%22%3A%22enwiki%22%2C%22event%22%3A%7B%22session_token%22%3A%227a881b31-4901-4183-b744-3d2c78fc290f%22%2C%22added%22%3Atrue%2C%22language%22%3A%22ru%22%2C%22time_spent%22%3A1%2C%22client_dt%22%3A%222018-06-18T11%3A42%3A06-0700%22%2C%22app_install_id%22%3A%220f4bd378-3204-4934-8fdf-1a206822ae87%22%7D%7D#011deployment-cache-te [16:56:05] xt04.deployment-prep.eqiad.wmflabs#0118582494#0112018-06-18T18:42:07#01170.187.174.34#011"WikipediaApp/2.7.234-dev-2018-06-18 (Android 8.1.0; Phone) Developer Channel" (Additional properties are not allowed (u'time_spent' was unexpected)) [16:57:12] bearloga: also i do not see the topic in kafka where valid events go [17:00:24] nuria_: thank you! two questions: (1) is client-side-00 for non-blacklisted schemas and is client-side-01 for blacklisted-from-mysql schemas? (2) what do you mean by topic in kafka? [17:00:39] bearloga: so (i think) either there was no valid event to date or kafka is run out of disk (it happens, this is labs) and cannot create new topic [17:00:46] bearloga: client-side-X is just for parallelizing stuff [17:00:50] so you need to look at both [17:00:57] bearloga: no, those just process events regardless of where they get stored after [17:01:23] nuria_: the event was sent using the wrong branch of the app, so the error is correct. will try again and check. [17:01:25] bearloga: the processors do not know anything about storage [17:01:29] bearloga: as for 'topic in kafka' [17:01:30] https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging/Architecture [17:01:36] (A little outdated but still mostly correct) [17:01:57] ah, okay [17:03:01] bearloga: ya, what it does not check out is that i would expect to find more errors from past events but i think there has been a restart in between and eventlogging on beta needs a restart to keep on functioning as it build s character whenit is been up for a while [17:04:52] nuria_: elukey from about an hour ago: "tried to restart eventlogging, some processors listed some kafka failures" [17:20:01] bearloga: ya, i would correct error, he restarted at 16:02 [17:20:17] bearloga: sorry, i would correct error and retry [17:33:45] nuria_: everything is looking good now! thank you so much for helping with this. this is new to me so I appreciate your patience! :) [17:38:24] (03CR) 10Nuria: [C: 04-1] "I do not think you need to deploy this to prod, npm run build should be sufficient. It is just that webpack here is not doing what the co" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/441037 (https://phabricator.wikimedia.org/T197482) (owner: 10Sahil505) [17:43:50] (03CR) 10Mforns: "> I do not think you need to deploy this to prod, npm run build" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/441037 (https://phabricator.wikimedia.org/T197482) (owner: 10Sahil505) [18:18:46] 10Analytics, 10Analytics-EventLogging, 10Readers-Web-Backlog: Some VirtualPageView are too long and fail EventLogging processing - https://phabricator.wikimedia.org/T196904#4303671 (10Jdlrobson) We had a chat about the different options here which were: 1. Trim all URLs to under a certain length ensuring we... [18:30:59] 10Analytics, 10Analytics-EventLogging, 10Readers-Web-Backlog: Some VirtualPageView are too long and fail EventLogging processing - https://phabricator.wikimedia.org/T196904#4303686 (10Ottomata) > We're a little concerned that limiting the source_url might not be enough. @mforns will be providing me a dump o... [18:57:30] 10Analytics, 10Operations, 10SRE-Access-Requests, 10Release-Engineering-Team (Kanban), 10User-greg: Requesting access for mbsantos - https://phabricator.wikimedia.org/T197237#4303725 (10RobH) a:05herron>03greg @greg: Would you review and approve/deny deployers access for @mbsantos? Once done, feel f... [19:01:13] 10Analytics-Tech-community-metrics, 10Code-Health, 10Release-Engineering-Team (Kanban): Develop canonical/single record of origin, machine readable list of all repos deployed to WMF sites. - https://phabricator.wikimedia.org/T190891#4303729 (10Aklapper) [19:01:34] 10Analytics, 10Operations, 10SRE-Access-Requests, 10Release-Engineering-Team (Kanban), 10User-greg: Requesting access for mbsantos - https://phabricator.wikimedia.org/T197237#4303730 (10greg) Sorry, approved! [19:01:55] 10Analytics, 10Operations, 10SRE-Access-Requests, 10Release-Engineering-Team (Kanban), 10User-greg: Requesting access for mbsantos - https://phabricator.wikimedia.org/T197237#4303731 (10RobH) a:05greg>03None [19:02:29] 10Analytics, 10Operations, 10SRE-Access-Requests: Requesting access for mbsantos - https://phabricator.wikimedia.org/T197237#4303732 (10greg) [19:04:22] jdlrobson, do you want me to copy the virtualpageview_errors.log anyware specific? Otherwise, you can get it under stat1004.eqiad.wmnet:/home/mforns/virtualpageview_errors.log [19:05:31] 10Analytics, 10Analytics-EventLogging, 10Readers-Web-Backlog: Some VirtualPageView are too long and fail EventLogging processing - https://phabricator.wikimedia.org/T196904#4303746 (10mforns) @Jdlrobson I put the virtualpageview error logs under ` stat1004.eqiad.wmnet:/home/mforns/virtualpageview_errors.log`... [19:41:24] 10Analytics, 10Analytics-Cluster, 10Patch-For-Review: Port Kafka clients to new jumbo cluster - https://phabricator.wikimedia.org/T175461#4303806 (10Ottomata) [19:54:30] !log removed Kafka MirrorMaker from kafka10(12|13|14) [19:54:34] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:45:41] 10Analytics, 10Analytics-Kanban, 10EventBus, 10ORES, and 4 others: Modify revision-score schema so that model probabilities won't conflict - https://phabricator.wikimedia.org/T197000#4303877 (10Halfak) Looks like we have some issues in the schema, but this is the access pattern we had in mind: https://ores... [20:46:19] 10Analytics, 10Analytics-EventLogging, 10Readers-Web-Backlog: Some VirtualPageView are too long and fail EventLogging processing - https://phabricator.wikimedia.org/T196904#4303878 (10Jdlrobson) @Ottomata that's true. Given the example error earlier, the EventLogging URLs I'm seeing are 2063-2176 code units... [20:47:19] 10Analytics, 10Analytics-Kanban, 10EventBus, 10ORES, and 4 others: Modify revision-score schema so that model probabilities won't conflict - https://phabricator.wikimedia.org/T197000#4303879 (10Ottomata) This is not long at all! I will submit a patch and attempt to just include possible model scores into... [20:47:42] 10Analytics, 10Analytics-Kanban, 10EventBus, 10ORES, and 3 others: Fix "score_schema" -- invalid JSON Schema - https://phabricator.wikimedia.org/T197828#4303880 (10Halfak) p:05Triage>03High [20:48:09] 10Analytics, 10Analytics-Kanban, 10EventBus, 10ORES, and 3 others: Fix "score_schema" -- invalid JSON Schema - https://phabricator.wikimedia.org/T197828#4303894 (10Halfak) Any other issues? [20:48:11] 10Analytics, 10Analytics-EventLogging, 10Page-Previews, 10Readers-Web-Backlog: Some VirtualPageView are too long and fail EventLogging processing - https://phabricator.wikimedia.org/T196904#4303895 (10Jdlrobson) 05stalled>03Open [20:48:31] 10Analytics, 10Analytics-EventLogging, 10Page-Previews, 10Readers-Web-Backlog: Some VirtualPageView are too long and fail EventLogging processing - https://phabricator.wikimedia.org/T196904#4272223 (10Jdlrobson) [20:55:04] 10Analytics, 10Analytics-Kanban, 10EventBus, 10ORES, and 3 others: Fix "score_schema" -- invalid JSON Schema - https://phabricator.wikimedia.org/T197828#4303902 (10Ottomata) OO true! ok [20:58:36] halfak: hey yt? [20:59:07] yes [21:00:36] hey just pinged you for dicsussion in #wikimedia-services [21:09:26] 10Analytics, 10Analytics-EventLogging, 10Page-Previews, 10Readers-Web-Backlog: Some VirtualPageView are too long and fail EventLogging processing - https://phabricator.wikimedia.org/T196904#4303915 (10mforns) @Jdlrobson @Ottomata Ha... I just thought that we might be ignoring lots of longer error logs...... [21:22:41] 10Analytics, 10Analytics-EventLogging, 10Page-Previews, 10Readers-Web-Backlog: Some VirtualPageView are too long and fail EventLogging processing - https://phabricator.wikimedia.org/T196904#4303926 (10mforns) @Jdlrobson Yea, many more errors when grepping for earlier fields... My bad. The new error dump i... [21:23:23] 10Analytics, 10EventBus, 10ORES, 10Patch-For-Review, and 3 others: Invalid field names in ORES models causing downstream Hive ingestion to fail - https://phabricator.wikimedia.org/T195979#4303927 (10Ottomata) Hey @Ladsgroup @awight ... https://ores.wikimedia.org/v3/scores/enwiki/?model_info=score_schema... [21:33:58] 10Analytics, 10Analytics-Kanban, 10EventBus, 10ORES, and 4 others: Modify revision-score schema so that model probabilities won't conflict - https://phabricator.wikimedia.org/T197000#4303970 (10Ottomata) Oh, this is not easy. The schema's have a few incompatible field names as noted in T197000. @Pchelol... [21:37:36] 10Analytics, 10Analytics-Kanban, 10EventBus, 10ORES, and 5 others: Emit revision-score event to EventBus and expose in EventStreams - https://phabricator.wikimedia.org/T167180#4303988 (10Ottomata) Due to the very harry problems in T195979 and T197000, I'm considering removing revision-score from EventStrea... [22:33:17] 10Analytics, 10Datasets-Archiving, 10Research: Make HTML dumps available - https://phabricator.wikimedia.org/T182351#3821170 (10Paladox) The migration to cloud was done i think now. [22:54:32] 10Analytics, 10Analytics-Wikimetrics, 10Story: Story: WikimetricsUser runs report against all wikis - https://phabricator.wikimedia.org/T70477#4304139 (10Liuxinyu970226) [22:58:49] 10Analytics, 10Analytics-Dashiki, 10Story: EEVSUser selects ALL wikis - https://phabricator.wikimedia.org/T70478#4304160 (10Liuxinyu970226)