[05:52:08] 10Analytics-Kanban, 10User-Elukey: dbstore1002 /srv filling up - https://phabricator.wikimedia.org/T168303#3464494 (10Marostegui) Its usual pace of 30G eaten per day... ``` [05:51:20] marostegui@dbstore1002:~$ df -hT /srv Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/tank-data xfs... [07:52:09] (03CR) 10DCausse: UDF for extracting primary full text search request (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/327855 (https://phabricator.wikimedia.org/T162054) (owner: 10EBernhardson) [10:22:44] hi, what version of kafka are we running? [10:24:44] dcausse: 0.9.0.1 [10:24:50] moritzm: thanks [11:55:51] * elukey lunch [12:26:10] anybody has any experience with http://ranger.apache.org/faq.html ? [12:26:12] looks nice [12:26:15] Cc: moritzm [12:31:59] I haven't, but sounds like something worth exploring/adding a task [12:32:28] just stumbled upon it while checking how to secure kafka [12:36:06] somewhat ironic that such a security component introduces new security issues :-) [12:36:08] grep "Apache Ranger" CVE/list | grep CVE-201[67] | wc -l [12:36:13] 6 [12:37:11] hahahah [12:37:28] but should be fine, mostly restriced to the UI (which would be restricted anyway) [12:37:29] the ranger is still learning [13:31:11] 10Analytics-Cluster, 10Analytics-Kanban: Unmount /a partition from 1002 - https://phabricator.wikimedia.org/T171373#3465906 (10Ottomata) Discovery jobs are just the ones that are puppetized, this is going to break some user jobs too [14:24:46] !log restarted mysql-eventbus eventlogging consumer with new consumer group [14:24:48] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:28:06] ottomata: o/ [14:32:51] hiii [14:55:55] 10Analytics-Cluster, 10Analytics-Kanban: Unmount /a partition from 1002 - https://phabricator.wikimedia.org/T171373#3466263 (10Nuria) >Discovery jobs are just the ones that are puppetized, this is going to break some user jobs too Let's try to see what those are before we dismount the drive, are those user cr... [14:58:26] 10Analytics-Cluster, 10Analytics-Kanban: Unmount /a partition from 1002 - https://phabricator.wikimedia.org/T171373#3466273 (10Ottomata) I'm not copying all /a to stat1005, but I'm copying most of it. There are some things that are redundant or obviously stale and not needed I've already copied non-user crea... [15:01:29] 10Analytics-Kanban, 10Contributors-Analysis, 10Patch-For-Review: Provide cumulative edit count in Data Lake edit data - https://phabricator.wikimedia.org/T161147#3466280 (10Nuria) Ping @Neil_P._Quinn_WMF [15:04:18] 10Analytics, 10Wikimedia-Mailing-lists: Delete eventlogging alerts e-mail list - https://phabricator.wikimedia.org/T170864#3466293 (10RobH) a:03RobH It seems there is no mailing list archives, so there isn't much to do except delete the list. I'll snag this and do so later today. [15:13:21] 10Analytics-EventLogging, 10Analytics-Kanban: ChangesListHighlights events missing from MySQL starting 2017-07-11 - https://phabricator.wikimedia.org/T170486#3466334 (10Nuria) [15:32:59] 10Analytics, 10cloud-services-team: Remove logging from labs for schema https://meta.wikimedia.org/wiki/Schema:CommandInvocation - https://phabricator.wikimedia.org/T166712#3305259 (10Andrew) Is this tagged with cloud-services-team in error, or is there something you need from us? [15:49:00] https://www.youtube.com/watch?v=2K4axNcmhvY --> next team building event [15:54:03] 10Analytics, 10Analytics-EventLogging: Run eventlogging purging script on beta labs to avoid disk getting full - https://phabricator.wikimedia.org/T171203#3466558 (10fdans) p:05Triage>03Normal [15:59:06] 10Analytics, 10EventBus, 10Wikimedia-Stream: Add tags to recentchange stream - https://phabricator.wikimedia.org/T171182#3466576 (10fdans) p:05Triage>03Low [16:02:01] 10Analytics-Cluster, 10Analytics-Kanban: CamusPartitionChecker does not work when topic names have '.' or '-' in them. - https://phabricator.wikimedia.org/T171099#3466584 (10fdans) [16:02:58] 10Analytics-Kanban, 10EventBus, 10Operations, 10Patch-For-Review, 10User-Elukey: Eventbus does not handle gracefully changes in DNS recursors - https://phabricator.wikimedia.org/T171048#3466588 (10fdans) [16:04:37] 10Analytics-EventLogging, 10Analytics-Kanban, 10Contributors-Analysis, 10DBA, and 2 others: Add index to mediawiki_page_create_1 table - https://phabricator.wikimedia.org/T170990#3466593 (10fdans) [16:05:09] 10Analytics-EventLogging, 10Analytics-Kanban, 10Contributors-Analysis, 10DBA, and 2 others: Add index to mediawiki_page_create_1 table - https://phabricator.wikimedia.org/T170990#3450014 (10fdans) a:03Ottomata [16:05:22] 10Analytics-Kanban, 10EventBus, 10Operations, 10Patch-For-Review, 10User-Elukey: Eventbus does not handle gracefully changes in DNS recursors - https://phabricator.wikimedia.org/T171048#3466596 (10fdans) a:03elukey [16:05:45] ottomata: ---^ are we going to deploy the fix ? [16:07:53] elukey: oh yeah! [16:08:01] we can do that [16:08:25] so Moritz will be unblocked [16:09:55] oh he's blocked? [16:10:13] elukey: you are welcome to deploy if you like, I'm happy to if you'd prefer if I do it [16:11:49] 10Analytics, 10Analytics-Wikistats: Wikistats2 bugs (4/4) - Detail page - https://phabricator.wikimedia.org/T170940#3466628 (10fdans) [16:12:20] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats2 bugs (4/4) - Detail page - https://phabricator.wikimedia.org/T170940#3448220 (10fdans) [16:18:23] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats2 bugs (4/4) - Detail page - https://phabricator.wikimedia.org/T170940#3466658 (10fdans) [16:19:24] ottomata: I'd let you do it if you don't mind since you know the service more than me from the code side [16:19:31] not sure if there are other pending things [16:19:48] Basically Moritz needs to reboot the other dns recursors for kernel updates [16:19:58] without causing eventbus outages :D [16:20:10] 10Analytics-Kanban: Error handling - https://phabricator.wikimedia.org/T171487#3466660 (10fdans) [16:20:40] k [16:20:47] 10Analytics, 10Analytics-Wikistats: Wikistats2 bugs (3/4) - Data issues - https://phabricator.wikimedia.org/T170937#3466673 (10fdans) [16:21:02] 10Analytics-Kanban: Error handling - https://phabricator.wikimedia.org/T171487#3466675 (10fdans) [16:21:18] 10Analytics-Kanban, 10Analytics-Wikistats: Error handling - https://phabricator.wikimedia.org/T171487#3466660 (10fdans) [16:21:34] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats2 bugs (3/4) - Data issues - https://phabricator.wikimedia.org/T170937#3448129 (10fdans) [16:21:43] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats2 bugs (3/4) - Data issues - https://phabricator.wikimedia.org/T170937#3448129 (10fdans) [16:24:24] 10Analytics, 10Analytics-Wikistats: Wikistats2 bugs (2/4) - Wiki selector - https://phabricator.wikimedia.org/T170936#3466688 (10fdans) [16:24:26] 10Analytics-Kanban, 10Analytics-Wikistats: Addition of Unique Devices metric - https://phabricator.wikimedia.org/T170461#3466687 (10fdans) [16:24:51] 10Analytics-Kanban, 10Analytics-Wikistats, 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban): Set up continuous integration for wikistats 2.0 UI - https://phabricator.wikimedia.org/T170458#3466692 (10mmodell) Are you using differential for code review? If so, I just need to set up the re... [16:25:27] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats2 bugs (2/4) - Wiki selector - https://phabricator.wikimedia.org/T170936#3448065 (10fdans) a:03fdans [16:29:49] 10Analytics, 10Analytics-Kanban: Wikistats2 bugs (1/4) - Dashboard and general UI - https://phabricator.wikimedia.org/T170933#3466702 (10fdans) a:03fdans [16:41:30] 10Analytics-Kanban, 10Analytics-Wikistats, 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban): Set up continuous integration for wikistats 2.0 UI - https://phabricator.wikimedia.org/T170458#3466734 (10mmodell) I've created the #meta-ci-jessie tag so CI should be happening now. [16:45:14] 10Analytics-EventLogging, 10Analytics-Kanban: Eventlogging: Ensure that `meta.request_id`'s auto MySQL index is not unique, while `meta.id` should be - https://phabricator.wikimedia.org/T171489#3466740 (10Ottomata) [16:51:03] 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: Eventlogging: Ensure that `meta.request_id`'s auto MySQL index is not unique, while `meta.id` should be - https://phabricator.wikimedia.org/T171489#3466779 (10Ottomata) [16:54:48] elukey: i just realized/remembered that we don't have auto-depool repool checks for scap eventbus deploys [16:54:51] we should add that! :) [16:54:56] i'll look into that... [16:55:06] would be nice to be able to just do scap deploy and have depools/repools done automaticallyed [16:55:11] for now, making som elunch... [16:55:28] ah yes! I've done it for AQS, should be very quick to do [16:57:05] https://github.com/wikimedia/analytics-aqs-deploy/commit/5917fd4b1ff448bda9a60f3e034f6ba3afc89f07 [16:57:30] ottomata: ===^ [16:58:42] probably just a matter of adding checks.yaml ? [17:05:03] ya elukey if you have a sec and want to add, would be much obliged [17:25:54] ottomata: attempt in https://gerrit.wikimedia.org/r/#/c/367447/2 [17:32:04] going afk! [17:32:06] * elukey off [17:45:58] yayyyyy we have CI in wikistats!! [17:51:57] laters! [17:51:58] thanks elukey! [18:13:32] fdans, \o/ [18:24:04] 10Analytics, 10EventBus: eventlogging-service-eventbus scap deployments should depool/pool during deployment - https://phabricator.wikimedia.org/T171506#3467218 (10Ottomata) [18:24:49] 10Analytics, 10EventBus, 10Scap: eventlogging-service-eventbus scap deployments should depool/pool during deployment - https://phabricator.wikimedia.org/T171506#3467239 (10Ottomata) [18:31:05] moritzm: FYI, i just deployed the statsd dns fixes to eventbus [18:31:10] so you should be unblocked therte [18:31:54] 10Analytics-Kanban, 10EventBus, 10Operations, 10Patch-For-Review, 10User-Elukey: Eventbus does not handle gracefully changes in DNS recursors - https://phabricator.wikimedia.org/T171048#3467266 (10Ottomata) Fix is deployed to eventbus. [19:07:44] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Unmount /a partition from 1002 - https://phabricator.wikimedia.org/T171373#3467429 (10Nuria) Talked to discovery analysts, seems that they have quite a bit important processing running under Mikhail's home directory, let's make sure those get trans... [19:38:07] (03PS3) 10Milimetric: Implement Wikistats metrics as Druid queries [analytics/refinery] - 10https://gerrit.wikimedia.org/r/365806 (https://phabricator.wikimedia.org/T170882) [19:49:27] 10Analytics-Kanban, 10Analytics-Wikistats: Define, Document (and test) Desktop and Mobile browser support for wikistats 2.0 - https://phabricator.wikimedia.org/T170457#3467673 (10mmodell) [19:49:29] 10Analytics-Kanban, 10Analytics-Wikistats, 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban): Set up continuous integration for wikistats 2.0 UI - https://phabricator.wikimedia.org/T170458#3467672 (10mmodell) 05Open>03Resolved [20:37:51] 10Analytics-Kanban, 10Patch-For-Review: Mediawiki History Druid indexing failed - https://phabricator.wikimedia.org/T170493#3467882 (10Nuria) Exception (from logs at /var/lib/druid/indexing-logs/) [{"wiki_db":"abwiki","event_entity":"revision","event_type":"create","event_timestamp":"2006-03-28 05:23:11.0","e... [20:39:55] 10Analytics-EventLogging, 10Analytics-Kanban: ChangesListHighlights events missing from MySQL starting 2017-07-11 - https://phabricator.wikimedia.org/T170486#3467886 (10Nuria) 05Open>03Resolved [20:40:13] 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: Ensure indexes are added to `meta_dt` and unique `meta_id` fields in eventbus MySQL tables in eventlogging databases. - https://phabricator.wikimedia.org/T170925#3467888 (10Nuria) 05Open>03Resolved [20:40:15] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Community-Tech, and 6 others: Record an event every time a new content namespace page is created - https://phabricator.wikimedia.org/T150369#3467889 (10Nuria) [20:40:24] 10Analytics-Kanban, 10Operations, 10ops-eqiad: Smartctl errors for one kafka1012 disk - https://phabricator.wikimedia.org/T168927#3467891 (10Nuria) [20:40:30] 10Analytics-Kanban, 10Operations, 10ops-eqiad: Smartctl errors for one kafka1012 disk - https://phabricator.wikimedia.org/T168927#3381297 (10Nuria) 05Open>03Resolved [20:40:46] 10Analytics-Kanban, 10Beta-Cluster-Infrastructure: deployment-eventlogging03 out of disk space - https://phabricator.wikimedia.org/T170522#3467893 (10Nuria) 05Open>03Resolved [20:40:59] 10Analytics-Kanban, 10Documentation, 10Services (watching): Document revision-create event for EventStreams - https://phabricator.wikimedia.org/T169245#3467895 (10Nuria) [20:41:06] 10Analytics-Kanban, 10Documentation, 10Services (watching): Document revision-create event for EventStreams - https://phabricator.wikimedia.org/T169245#3391503 (10Nuria) 05Open>03Resolved [21:45:09] 10Analytics, 10EventBus, 10Services (later): Document ChangeProp and EventBus monitoring - https://phabricator.wikimedia.org/T171533#3468153 (10Pchelolo) [21:45:38] 10Analytics, 10ChangeProp, 10EventBus, 10Epic, and 2 others: [EPIC] Develop a JobQueue backend based on EventBus - https://phabricator.wikimedia.org/T157088#3468168 (10Pchelolo) [21:45:39] 10Analytics, 10EventBus, 10Services (later): Document ChangeProp and EventBus monitoring - https://phabricator.wikimedia.org/T171533#3468167 (10Pchelolo) [21:56:26] 10Analytics-Cluster, 10Analytics-Kanban, 10Language-Team, 10MediaWiki-extensions-UniversalLanguageSelector, and 3 others: Migrate table creation query to oozie - https://phabricator.wikimedia.org/T170764#3468188 (10Milimetric) priorities have sadly shifted a bit, I may not be able to get to this. cc @nuria.