[01:11:55] <wikibugs>	 10Analytics, 10Event-Platform, 10Inuka-Team: Create KaiOS error stream - https://phabricator.wikimedia.org/T250177 (10SBisson) 05Open→03Resolved a:03Ottomata The stream is in place and events have started appearing in the dashboard. Thanks!
[01:12:00] <wikibugs>	 10Analytics, 10Event-Platform, 10Inuka-Team (Kanban), 10KaiOS-Wikipedia-app (MVP), 10Patch-For-Review: Capture and send back client-side errors - https://phabricator.wikimedia.org/T248615 (10SBisson)
[01:50:18] <wikibugs>	 10Analytics, 10Event-Platform, 10Inuka-Team: Create KaiOS error stream - https://phabricator.wikimedia.org/T250177 (10Ottomata) Glad it is working!
[02:18:08] <milimetric>	 hey, if anyone's around it looks like webrequest load is stuck waiting for _IMPORTED flags https://hue.wikimedia.org/oozie/list_oozie_coordinator/0099569-200303081945184-oozie-oozi-C/
[02:26:09] <ottomata>	 milimetric:  yoohoo
[02:26:11] <ottomata>	 looking
[02:26:18] <milimetric>	 right, looks like camus didn't write the _IMPORTED flag
[02:26:31] <milimetric>	 hdfs dfs -ls -t /wmf/data/raw/webrequest/webrequest_text/hourly/2020/04/30/18 has it
[02:26:32] <ottomata>	 i changed some camus configs today but it was supposed to be a no op
[02:26:34] <milimetric>	 hdfs dfs -ls -t /wmf/data/raw/webrequest/webrequest_text/hourly/2020/04/30/19 doesn;t
[02:27:54] <milimetric>	 same for upload, 19 is the first hour that doesn't have it
[02:28:13] <ottomata>	 that liines up with when i merged the change
[02:28:16] <milimetric>	 data all looks like it's there... but there's something weird with the filenames, the new ones are shorter...?
[02:30:07] <milimetric>	 nvm, it's just because the flags are owned by analytics_private_data, just looked weird
[02:30:48] <ottomata>	 AH yes 
[02:30:54] <ottomata>	 lost a param in the refactor
[02:30:57] <ottomata>	 and misised it in rview
[02:32:29] <milimetric>	 this one right? https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/593288/
[02:32:50] <ottomata>	 yup
[02:32:51] <ottomata>	 check                 => $monitoring_enabled,
[02:32:56] <ottomata>	 should have been moved into the Camus::Job defaults
[02:33:22] <ottomata>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/593659/1/modules/profile/manifests/analytics/refinery/job/camus.pp
[02:33:39] <milimetric>	 you think we can just write _IMPORTED flags and rerun the instances that timed out?
[02:34:03] <milimetric>	 or would that affect the actual data too?'
[02:34:29] <ottomata>	 i think we can rerun the checker for each hour
[02:36:14] <ottomata>	 so our camus checker cli wrapper doesn't allow us to specifiy a range
[02:36:23] <ottomata>	 but we can script up a little thing to iterate over the missed hours
[02:36:47] <ottomata>	 if you want to help dan you could generate a list of the missed hours that looks like
[02:36:47] <ottomata>	 yyyy-mm-dd-HH-MM-SS
[02:37:04] <ottomata>	 i thnk only webrequest topics are affected here
[02:37:21] <ottomata>	 they are the only ones that use the checker's _IMPORTED 
[02:37:21] <ottomata>	 flag
[02:37:30] <ottomata>	 but don't worry about it 
[02:37:32] <ottomata>	 i can do it
[02:37:34] <ottomata>	 i broke it :)
[02:38:15] <milimetric>	 it's all good, I'm up anyway, I can slowpoke look over everything.  So that check parameter was covered here: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/593612/1/modules/camus/manifests/job.pp
[02:38:36] <milimetric>	 just the $monitoring_enabled wasn't passed anywhere
[02:40:52] <milimetric>	 ottomata:
[02:40:54] <milimetric>	 https://www.irccloud.com/pastebin/x5CMDNBh/
[02:41:24] <ottomata>	 its just that check in camus job defaults to undef
[02:41:37] <ottomata>	 and refinery::job::camus was previously settting it to true manually for webrequest
[02:41:41] <ottomata>	 and other jobs
[02:41:49] <ottomata>	 thx
[02:41:53] <milimetric>	 that's all the hours I see folders for in hdfs 
[02:43:10] <milimetric>	 though, ottomata, 2020-05-01-02 doesn't look done yet
[02:43:47] <ottomata>	 probably will need checked
[02:45:34] <milimetric>	 why is there a little lowercase "y" at the beginning of this file? https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/593288/7/modules/camus/templates/webrequest.erb (I mean, it was deleted but it's weird)
[02:46:25] <ottomata>	 i think it was some typo in the old template
[02:46:29] <ottomata>	 that somehow got there
[02:46:34] <ottomata>	 those templates were really messy
[02:47:07] <milimetric>	 yeah, it's weird and it's in the webrequest one, just wondering if it could've covered up some other problem that got uncovered by the refactor
[02:54:05] <ottomata>	 hm ya
[02:59:26] <ottomata>	 ok milimetric all _IMPORTED flags back
[02:59:32] <ottomata>	 if you wanna double check
[02:59:38] <milimetric>	 will do
[03:00:21] <milimetric>	 ottomata: I should still restart 19 and 20 https://hue.wikimedia.org/oozie/list_oozie_coordinator/0099569-200303081945184-oozie-oozi-C/
[03:00:53] <ottomata>	 ya
[03:06:16] <ottomata>	 looking ok now ya milimetric ?
[03:06:38] <milimetric>	 looks ok ottomata, I reran the 4 that had errored out, I'll send a follow-up to my email and look again tomorrow
[03:09:02] <wikibugs>	 (03PS1) 10Lex Nasser: Add editors by country data to AQS [analytics/aqs] - 10https://gerrit.wikimedia.org/r/593660 (https://phabricator.wikimedia.org/T238365)
[03:09:46] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add editors by country data to AQS [analytics/aqs] - 10https://gerrit.wikimedia.org/r/593660 (https://phabricator.wikimedia.org/T238365) (owner: 10Lex Nasser)
[03:11:45] <ottomata>	 ok
[03:12:13] <ottomata>	 gonna sign off text me if you need anything else
[03:12:15] <ottomata>	 goodnighters!
[03:12:17] <ottomata>	 thank you!
[03:13:05] <wikibugs>	 (03CR) 10Lex Nasser: "I think this patchset covers everything but the test files. Build fails unfortunately. Handing this off to milimetric." [analytics/aqs] - 10https://gerrit.wikimedia.org/r/593660 (https://phabricator.wikimedia.org/T238365) (owner: 10Lex Nasser)
[03:15:09] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Add editors per country data to AQS API  (geoeditors) - https://phabricator.wikimedia.org/T238365 (10lexnasser) a:05lexnasser→03Milimetric Handing the remainder of this task off to Dan.  The jobs for which this endpoint relies are tentatively complete.
[07:29:58] <wikibugs>	 10Analytics, 10Dumps-Generation: page_restrictions field incomplete in current and historical dumps - https://phabricator.wikimedia.org/T251411 (10ArielGlenn) This has been broken a long time. I need to check a little bit more of the history, but I can verify that exports use the contents of the obsolete page_...
[07:34:26] <wikibugs>	 10Analytics, 10Dumps-Generation, 10Patch-For-Review: page_restrictions field incomplete in current and historical dumps - https://phabricator.wikimedia.org/T251411 (10ArielGlenn) See also: T218446
[07:35:31] <wikibugs>	 10Analytics, 10Dumps-Generation, 10Core Platform Team Workboards (Clinic Duty Team), 10Patch-For-Review: page_restrictions field incomplete in current and historical dumps - https://phabricator.wikimedia.org/T251411 (10ArielGlenn)
[09:28:22] <elukey>	 nothing seems to be on fire anymore
[09:38:48] <wikibugs>	 (03PS2) 10Elukey: Release upstream version 0.36.0 [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/591440 (https://phabricator.wikimedia.org/T249495)
[11:41:02] <wikibugs>	 10Analytics, 10LDAP-Access-Requests, 10Operations: LDAP access to the wmf group for Antonino Hemmer (superset, turnilo, hue) - https://phabricator.wikimedia.org/T251123 (10Dzahn) @DZierten Could you also give as an expiration date please? For contractors we need to add that to the repository on our side. Tha...
[12:07:07] <wikibugs>	 10Analytics, 10Dumps-Generation, 10Core Platform Team Workboards (Clinic Duty Team), 10Patch-For-Review: page_restrictions field incomplete in current and historical dumps - https://phabricator.wikimedia.org/T251411 (10daniel) @ArielGlenn since you are currently part of the Clinic Duty team, I'm moving thi...
[12:08:14] <wikibugs>	 10Analytics, 10Dumps-Generation, 10Core Platform Team Workboards (Clinic Duty Team), 10Patch-For-Review: page_restrictions field incomplete in current and historical dumps - https://phabricator.wikimedia.org/T251411 (10daniel) p:05Triage→03Low tagging as low, since nobody noticed for a long time - so I...
[12:13:49] <wikibugs>	 10Analytics: check leftovers of jmorgan - https://phabricator.wikimedia.org/T251600 (10elukey)
[12:14:25] <wikibugs>	 10Analytics: check leftovers of jmorgan - https://phabricator.wikimedia.org/T251600 (10elukey) @leila whenever you have time, same request to check files left :)
[12:15:00] <elukey>	 :( :( :( --^
[13:13:22] <wikibugs>	 10Analytics, 10Dumps-Generation, 10Core Platform Team Workboards (Clinic Duty Team), 10Patch-For-Review: page_restrictions field incomplete in current and historical dumps - https://phabricator.wikimedia.org/T251411 (10Tgr) >>! In T251411#6099487, @ArielGlenn wrote: > The page_restrictions table is already...
[13:17:09] <wikibugs>	 10Analytics, 10Fundraising-Backlog, 10fundraising-tech-ops: Install superset on front end server for analytics - https://phabricator.wikimedia.org/T245755 (10Jgreen) After a bunch of refactoring and testing, I've managed to refactor the Analytics superset deploy project and puppet code for the frack environm...
[14:22:14] <wikibugs>	 10Analytics, 10Operations: systemd::syslog conf should use :programname equals instead of startswith - https://phabricator.wikimedia.org/T251606 (10Ottomata)
[14:59:06] <wikibugs>	 10Analytics, 10LDAP-Access-Requests, 10Operations: LDAP access to the wmf group for Antonino Hemmer (superset, turnilo, hue) - https://phabricator.wikimedia.org/T251123 (10Nuria) I think Antonio is a full time employee. @DZierten to confirm
[15:41:46] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform: Automate ingestion and refinement into Hive of event data from Kafka - https://phabricator.wikimedia.org/T251609 (10Ottomata)
[15:44:15] <wikibugs>	 10Analytics, 10LDAP-Access-Requests, 10Operations: LDAP access to the wmf group for Antonino Hemmer (superset, turnilo, hue) - https://phabricator.wikimedia.org/T251123 (10DZierten) He is a full time employee.  Thanks for checking.  Do you know where I can change his status?  Thanks
[15:57:13] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform: Automate ingestion and refinement into Hive of event data from Kafka - https://phabricator.wikimedia.org/T251609 (10Ottomata) If we continue to use Camus, the ideal state of things would look something like:  - A single Camus uses...
[16:06:51] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform: Automate ingestion and refinement into Hive of event data from Kafka - https://phabricator.wikimedia.org/T251609 (10Ottomata)
[16:10:01] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform: Automate ingestion and refinement into Hive of event data from Kafka - https://phabricator.wikimedia.org/T251609 (10Ottomata)
[16:29:37] <wikibugs>	 10Analytics: check leftovers of jmorgan - https://phabricator.wikimedia.org/T251600 (10leila) @elukey I'm double-checking with Jonathan and will get back to you. :)
[16:29:49] <wikibugs>	 10Analytics: check leftovers of jmorgan - https://phabricator.wikimedia.org/T251600 (10leila) a:03leila
[16:55:23] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Event-Platform, 10CPT Initiatives (Modern Event Platform (TEC2)), 10MW-1.34-notes (1.34.0-wmf.20; 2019-08-27): Refactor EventBus mediawiki configuration - https://phabricator.wikimedia.org/T229863 (10Pchelolo) So, one thing that's touching the code around it is: h...
[16:59:51] <elukey>	 !log test prometheus-druid-exporter 0.8 on druid1001 (deb packages not yet uploaded, just build and manually installed)
[16:59:53] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[18:44:08] <wikibugs>	 10Analytics, 10Fundraising-Backlog, 10fundraising-tech-ops: Install superset on front end server for analytics - https://phabricator.wikimedia.org/T245755 (10Jgreen) a:03Jgreen I worked up a local patch for werkzeug security.py which will hopefully allow us to move forward on this project until an upstream...