[03:18:47] <wikibugs>	 10Analytics-Kanban, 10Patch-For-Review: Unique druid segment compaction - https://phabricator.wikimedia.org/T197885#4305708 (10Nuria) These changed do not affect the availability of daily data.
[06:24:30] <wikibugs>	 (03CR) 10Elukey: [V: 032 C: 032] Scap: Remove git_server from scap.cfg [analytics/pivot/deploy] - 10https://gerrit.wikimedia.org/r/441236 (https://phabricator.wikimedia.org/T162814) (owner: 10Thcipriani)
[06:24:46] <wikibugs>	 (03CR) 10Elukey: [V: 032 C: 032] Scap: Remove git_server from scap.cfg [analytics/turnilo/deploy] - 10https://gerrit.wikimedia.org/r/441239 (https://phabricator.wikimedia.org/T162814) (owner: 10Thcipriani)
[06:26:08] <elukey>	 hello hello :)
[06:35:39] <wikibugs>	 10Analytics, 10Cleanup, 10Operations, 10Patch-For-Review: Archive operations/puppet/varnishkafka repository - https://phabricator.wikimedia.org/T197503#4310264 (10elukey)
[06:35:52] <joal>	 Hi elukey - I'm not here today (as youcan read)
[06:36:53] <elukey>	 :)
[06:37:01] <elukey>	 hello joal !
[06:37:45] <joal>	 Thanks a lot elukey for you messages from Prague - They were really super informative :)
[06:39:41] <elukey>	 thanks! 
[06:41:07] <elukey>	 there's a ton of work for the next months :D
[06:42:05] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10User-Elukey: Import some Analytics git puppet submodules to operations/puppet - https://phabricator.wikimedia.org/T188377#4310268 (10elukey)
[06:42:08] <wikibugs>	 10Analytics, 10Cleanup, 10Operations, 10Patch-For-Review: Archive operations/puppet/varnishkafka repository - https://phabricator.wikimedia.org/T197503#4310267 (10elukey)
[06:45:23] <elukey>	 I'd also be interested, long term, to test Apache bigtop
[06:48:21] <joal>	 elukey: I've seen that - I don't know how mature the thing is, but it'd be great (even maybe contribute?)
[06:54:31] <elukey>	 joal: I suspiciously saw a ton of "bigtop" references in our init.d files here and there, so atm I think that cdh is based on that
[06:54:37] <elukey>	 but it is only a speculation :)
[06:58:10] <joal>	 :)
[08:44:17] <elukey>	 re-done the varnishkafka dashboard https://grafana.wikimedia.org/dashboard/db/varnishkafka
[08:44:44] <elukey>	 still not perfect but we can now "zoom-in" for each host
[08:45:11] <elukey>	 very interesting thing happened a while ago to cp5012 and another cp host, namely this
[08:45:14] <elukey>	 https://grafana.wikimedia.org/dashboard/db/varnishkafka?orgId=1&from=now-30d&to=now&var-instance=eventlogging&var-host=cp5012
[08:45:28] <elukey>	 I found it only by chance while looking at the dashboard (next step - alarms)
[08:54:12] <elukey>	 the recovery for cp5012 was me restarting the varnishkafka eventlogging instance
[08:54:38] <elukey>	 but the "start" of the errors seems to be related to kafka-jumbo1004 latency dropping to zero
[09:15:41] <elukey>	 err kafka-jumbo1005 sorry
[09:15:49] <elukey>	 on cp5012 everything started with
[09:15:50] <elukey>	 Jun 11 15:48:16 cp5012 varnishkafka[44148]: KAFKADR: Kafka message delivery error: Local: Message timed out
[09:16:01] <elukey>	 and I can see a Jun 11 15:48:16 cp5012 varnishkafka[44148]: KAFKAERR: Kafka error (-192): ssl://kafka-jumbo1005.eqiad.wmnet:9093/1005: 4 request(s) timed out: disconnect
[09:17:04] <elukey>	 in theory the delivery callback is called if either the msg is delivered or when it failed after 3 retries
[09:17:51] <elukey>	 and local message timeout seems to be what we were seeing on mirror maker, namely messages queued in the librdkafka local queue and expiring before getting delivered 
[09:18:07] <elukey>	 we time out after 5 mins of being in the queue 
[09:19:18] <elukey>	 the theory is that if a queue is overwhelmed, expecially due to a error condition that triggers retries, then it fills up and it fails to drain the backlog in time before the first message expired
[09:19:30] <elukey>	 then it seems that it keeps going like this until a restart is issued
[09:37:35] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10User-Elukey: Varnishkafka eventlogging instances delivery failures - https://phabricator.wikimedia.org/T198070#4310724 (10elukey) p:05Triage>03High
[09:48:33] <wikibugs>	 10Analytics, 10Operations, 10SRE-Access-Requests: Requesting access for mbsantos - https://phabricator.wikimedia.org/T197237#4310779 (10faidon) Yes, let's not block this for yet another week! Consider this approved, please go ahead.
[10:10:01] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Varnishkafka eventlogging instances delivery failures - https://phabricator.wikimedia.org/T198070#4310859 (10elukey) As part of this task I refactored https://grafana.wikimedia.org/dashboard/db/varnishkafka to better visualize per host met...
[10:35:26] * elukey lunch! (bb in ~2h)
[11:56:18] <wikibugs>	 (03PS1) 10Jonas Kress (WMDE): Track new API reported maxlag for wikidata on grafana dashboard [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/441838
[11:56:23] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Track new API reported maxlag for wikidata on grafana dashboard [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/441838 (owner: 10Jonas Kress (WMDE))
[11:56:43] <wikibugs>	 (03PS2) 10Jonas Kress (WMDE): Track new API reported maxlag for wikidata on grafana dashboard [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/441838
[11:58:07] <wikibugs>	 (03PS3) 10Jonas Kress (WMDE): Track new API reported maxlag for wikidata on grafana dashboard [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/441838
[12:07:37] <wikibugs>	 (03PS4) 10Addshore: Track new API reported maxlag for wikidata on grafana dashboard [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/441838 (https://phabricator.wikimedia.org/T196868) (owner: 10Jonas Kress (WMDE))
[12:07:45] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Track new API reported maxlag for wikidata on grafana dashboard [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/441838 (https://phabricator.wikimedia.org/T196868) (owner: 10Jonas Kress (WMDE))
[12:10:50] <wikibugs>	 (03CR) 10Addshore: [C: 04-1] Track new API reported maxlag for wikidata on grafana dashboard (031 comment) [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/441838 (https://phabricator.wikimedia.org/T196868) (owner: 10Jonas Kress (WMDE))
[12:43:37] <wikibugs>	 10Analytics, 10Operations, 10hardware-requests: eqiad: (1) new stat box to offload users from stat1005 - https://phabricator.wikimedia.org/T196345#4311561 (10faidon) a:05elukey>03RobH That spare assignment sounds good to me, consider it approved. @RobH, you can go ahead :)
[12:45:07] <elukey>	 stat1007 coming --^
[12:46:19] <wikibugs>	 10Analytics, 10Operations: Broken apt config on kafka/analytics hosts - https://phabricator.wikimedia.org/T198092#4311574 (10MoritzMuehlenhoff) p:05Triage>03High
[12:47:23] <elukey>	 moritzm: o/ on it --^
[12:48:47] <moritzm>	 ack! I'm afk for an hour, but happy to review patches afterwards
[13:00:46] <wikibugs>	 10Analytics: Add a safe failover for analytics1003 - https://phabricator.wikimedia.org/T198093#4311598 (10elukey) p:05Triage>03Normal
[13:31:00] <wikibugs>	 10Analytics, 10Cleanup, 10Operations, 10User-Elukey: Archive operations/puppet/jmxtrans repository - https://phabricator.wikimedia.org/T198097#4311727 (10elukey) p:05Triage>03Low
[13:31:26] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Operations, 10User-Elukey: Archive operations/puppet/kafkatee repository - https://phabricator.wikimedia.org/T198098#4311739 (10elukey) p:05Triage>03Low
[13:47:50] <elukey>	 ottomata: o/
[13:48:05] <ottomata>	 hiii
[13:49:10] <ottomata>	 elukey:  sounds like the offsite was pretty productive!
[13:51:24] <elukey>	 it was! A lot of things to chat and discuss
[13:59:56] <wikibugs>	 10Analytics: Add a safe failover for analytics1003 - https://phabricator.wikimedia.org/T198093#4311928 (10Ottomata) > there might be a chance that the snapshot used in a restore emergency operation leads to a corrupted database Is there?  We don't stop Mariadb,but mylvmbackup locks the tables (and flushes writes...
[14:00:43] <ottomata>	 Ah i missed those other mIrromakr alerts, thanks elukey!
[14:01:17] <elukey>	 np!
[14:02:54] <wikibugs>	 10Analytics: Add a safe failover for analytics1003 - https://phabricator.wikimedia.org/T198093#4311955 (10elukey) >>! In T198093#4311928, @Ottomata wrote: >> there might be a chance that the snapshot used in a restore emergency operation leads to a corrupted database > Is there?  We don't stop Mariadb,but mylvmb...
[14:03:21] <wikibugs>	 10Analytics: Add a safe failover for analytics1003 - https://phabricator.wikimedia.org/T198093#4311972 (10Ottomata) Hm!  interesting.
[14:04:53] <elukey>	 ottomata: ---^ this is my n00b understanding of the issue, it might not be the case for our set up, but I thought to add a note to discuss about it
[14:05:34] <ottomata>	 elukey:  ya sounds good!
[14:06:43] <wikibugs>	 10Analytics: Add a safe failover for analytics1003 - https://phabricator.wikimedia.org/T198093#4311980 (10Marostegui) That is correct. It might or might not work. MariaDB will go thru a normal InnoDB recovery process (like if it had crashed). So there are chances that it might work, but it can also end up with c...
[14:16:42] <wikibugs>	 (03CR) 10Mforns: [V: 032 C: 032] "Nuria, we tested this in the canary and everything worked well. So, I'm merging this. Thanks anyway for the review." [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/441037 (https://phabricator.wikimedia.org/T197482) (owner: 10Sahil505)
[14:19:54] <wikibugs>	 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10Goal, and 3 others: FY17/18 Q4 Program 8 Services Goal: Complete the JobQueue transition to EventBus - https://phabricator.wikimedia.org/T190327#4312057 (10Pchelolo)
[14:19:58] <wikibugs>	 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10Notifications, and 2 others: Make EchoNotification job JSON-serializable - https://phabricator.wikimedia.org/T192945#4312055 (10Pchelolo) 05Open>03Resolved
[14:25:27] <moritzm>	 elukey: thanks, patch is working fine, apt-get update works again
[14:26:06] <elukey>	 moritzm: np! I just realized that the patch is wrong, I am amending it :P
[14:29:36] <moritzm>	 ah, indeed. I had only tested the jessie code path :-)
[14:30:07] <elukey>	 I usually run pcc but this time I was super confident (last famous words)
[14:36:00] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review: Broken apt config on kafka/analytics hosts - https://phabricator.wikimedia.org/T198092#4312118 (10elukey)
[14:36:10] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review: Broken apt config on kafka/analytics hosts - https://phabricator.wikimedia.org/T198092#4311563 (10elukey)
[14:56:32] <wikibugs>	 10Analytics, 10Cleanup, 10Operations, 10Patch-For-Review: Archive operations/puppet/varnishkafka repository - https://phabricator.wikimedia.org/T197503#4312224 (10Krinkle)
[14:57:17] <wikibugs>	 10Analytics, 10Cleanup, 10Operations, 10Patch-For-Review: Archive operations/puppet/varnishkafka repository - https://phabricator.wikimedia.org/T197503#4294420 (10Krinkle) At <https://gerrit.wikimedia.org/r/#/admin/projects/operations/puppet/varnishkafka>, I've set the description to `[ARCHIVED] Merged int...
[15:03:43] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review: Broken apt config on kafka/analytics hosts - https://phabricator.wikimedia.org/T198092#4312269 (10Nuria) a:03elukey
[15:27:28] <wikibugs>	 10Analytics, 10Operations, 10hardware-requests: eqiad: (1) new stat box to offload users from stat1005 - https://phabricator.wikimedia.org/T196345#4312330 (10elukey) Sorry to cause all this noise @faidon and @RobH, but after a chat with my team we have some concern related to moving people around between sta...
[15:27:53] <wikibugs>	 10Analytics, 10Operations, 10hardware-requests: eqiad: (1) new stat box to offload users from stat1005 - https://phabricator.wikimedia.org/T196345#4312331 (10elukey) So to summarize: my team would prefer a new host rather than the spare one.
[15:31:45] <wikibugs>	 10Analytics, 10Cleanup, 10Operations, 10Patch-For-Review, 10User-Elukey: Archive operations/puppet/jmxtrans repository - https://phabricator.wikimedia.org/T198097#4312344 (10elukey)
[15:32:18] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10User-Elukey: Archive operations/puppet/kafkatee repository - https://phabricator.wikimedia.org/T198098#4312347 (10elukey)
[15:35:01] <wikibugs>	 10Analytics, 10Operations, 10hardware-requests: eqiad: (1) new stat box to offload users from stat1005 - https://phabricator.wikimedia.org/T196345#4312349 (10RobH) We don't have to move users off a box as soon as the warranty expires, in fact we tend to run boxes for 4-5 years when warrantied for 3.  @elukey...
[15:36:25] <nuria_>	 ping elukey 
[15:43:41] <wikibugs>	 10Analytics, 10Operations, 10hardware-requests: eqiad: (1) new stat box to offload users from stat1005 - https://phabricator.wikimedia.org/T196345#4312357 (10Ottomata) We have budget for a new stat box next FY.  We'd like to use that budget to order the new box, move stat1005 users to it, and then use stat10...
[15:51:48] <wikibugs>	 10Analytics, 10Operations, 10SRE-Access-Requests: Requesting access for mbsantos - https://phabricator.wikimedia.org/T197237#4312386 (10RobH)
[16:08:32] <wikibugs>	 10Analytics, 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Requesting access for mbsantos - https://phabricator.wikimedia.org/T197237#4312438 (10RobH) 05Open>03Resolved @MSantos:   Your access request has been merged live, with all of the groups you requested.  Since this is a new account fo...
[16:21:56] <wikibugs>	 10Analytics, 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Requesting access for mbsantos - https://phabricator.wikimedia.org/T197237#4312472 (10MSantos) Thanks, for all the support and thank you @RobH for the warnings.
[16:49:31] <wikibugs>	 10Analytics, 10Analytics-Wikistats: Correct spelling mistake in description meta tag of wikistats 2 - https://phabricator.wikimedia.org/T198122#4312608 (10sahil505)
[16:50:39] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Correct spelling mistake in description meta tag of wikistats 2 - https://phabricator.wikimedia.org/T198122#4312608 (10sahil505) p:05Triage>03Normal
[17:01:05] <wikibugs>	 (03PS1) 10Sahil505: Corrected spelling mistakes in description meta tag [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/441917 (https://phabricator.wikimedia.org/T198122)
[17:01:45] <wikibugs>	 (03CR) 10Sahil505: [C: 031] Corrected spelling mistakes in description meta tag [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/441917 (https://phabricator.wikimedia.org/T198122) (owner: 10Sahil505)
[17:12:02] <wikibugs>	 10Analytics, 10Operations, 10hardware-requests: eqiad: (1) new stat box to offload users from stat1005 - https://phabricator.wikimedia.org/T196345#4312713 (10elukey) >>! In T196345#4312349, @RobH wrote: > We don't have to move users off a box as soon as the warranty expires, in fact we tend to run boxes for...
[17:29:12] <wikibugs>	 10Analytics, 10Cleanup, 10Operations, 10Patch-For-Review, 10User-Elukey: Archive operations/puppet/jmxtrans repository - https://phabricator.wikimedia.org/T198097#4312769 (10elukey)
[17:29:35] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10User-Elukey: Archive operations/puppet/kafkatee repository - https://phabricator.wikimedia.org/T198098#4312770 (10elukey)
[17:30:20] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10User-Elukey: Import some Analytics git puppet submodules to operations/puppet - https://phabricator.wikimedia.org/T188377#4312793 (10elukey) All modules imported into operations/puppet, the remaining thing to do is cleaning up (subtasks).
[17:46:55] <ottomata>	 wow, hm, milimetric do you know about RecentChanges category changes?
[17:47:08] <ottomata>	 i just noticed that those are on the Special:RecentChanges page
[17:47:16] <ottomata>	 do you have any idea how those get there?
[17:47:24] <ottomata>	 does Mediawiki know about the category changes?
[17:51:25] <James_F>	 ottomata: It's shallow (non-recursive) category changes.
[17:52:08] * elukey off!
[17:53:02] <James_F>	 ottomata: See https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/refs/heads/master/includes/changes/CategoryMembershipChange.php which triggers newForCategorization().
[18:05:37] <ottomata>	 James_F:  where does the $categoryTitle come from?
[18:06:32] <ottomata>	 oh i found it
[18:06:34] <ottomata>	 its from a job
[18:06:41] <ottomata>	 which gets it out of the content
[18:06:43] <ottomata>	 hmmm
[18:06:49] <James_F>	 ottomata: From https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/refs/heads/master/includes/jobqueue/jobs/CategoryMembershipChangeJob.php
[18:06:51] <James_F>	 Yeah.
[18:07:02] <James_F>	 So it's not a perfectly reliable stream.
[18:07:07] <James_F>	 But it's pretty good.
[18:07:23] <ottomata>	 James_F:  in that it only includes the categories for the revision of that one page?  
[18:08:01] <ottomata>	 oh wow, and this job is triggered by recentchanges table?
[18:08:04] <James_F>	 In that if I add a category and remove it, and the jobqueue de-dupes the jobs before the first executes, it'll never get flagged as ever having been added?
[18:08:13] <James_F>	 And because sometimes jobs just go missing. :-(
[18:08:19] <ottomata>	 right
[18:08:43] <James_F>	 Maybe I'm too pessimistic about jobs; nowadays they're pretty reliable.
[18:08:48] <ottomata>	 their better for sure. hm
[18:10:18] <ottomata>	 this job is querying the recentchange table for the timestamp of the last category change for a page
[18:11:01] <ottomata>	 then gets all revisions since then
[18:11:09] <ottomata>	 wow
[18:11:20] <ottomata>	 gets the content for each on
[18:11:23] <ottomata>	 e
[18:11:33] <ottomata>	 they're*
[18:12:00] <ottomata>	 Pchelolo: ^ could that be done via change-prop?
[18:12:39] <ottomata>	 instead of querying all revisions since last update, insert job to emit category change for a revision if it has a category change?
[18:13:26] <Pchelolo>	 James_F: nowadays jobs never go missing :)
[18:13:28] <ottomata>	 it'd be cool to have a page-category-change stream
[18:13:42] <ottomata>	 maybe we could even just make a hook in CategoryMembership change
[18:13:44] <ottomata>	 and not change hte job at all
[18:13:49] <ottomata>	 then the job could emit an event to eventbus
[18:13:50] <Pchelolo>	 ottomata: what exactly needs to be done via change-prop?
[18:13:58] <James_F>	 Pchelolo: Uh-huh. ;-)
[18:14:27] <ottomata>	 Pchelolo:  i'm just noticing that this job kinda does a batch version of what a lot of changeprop does
[18:14:44] <ottomata>	 job runs, gets all revisions since last time, parsese the content of each rev for categories, then inserts into recentchange table
[18:15:07] <ottomata>	 probably  not worth the effort to do that on change prop based on revision-create events, but it could be done
[18:15:16] <ottomata>	 i really just think it'd be cool to have a category-change stream
[18:15:33] <ottomata>	 which coudl be done without any changes to the job
[18:15:57] <ottomata>	 wow, and James_F there are RecentChange entries for each category add and remove?
[18:16:11] <James_F>	 ottomata: Yes.
[18:16:49] <James_F>	 Though as I said, it's only direct membership. The real magic would be something that understand the whole category graph and charts sub-membership changes, but we don't have US$bns to spend. :-)
[18:17:15] <ottomata>	 aye
[18:17:33] <ottomata>	 so maybe a direct cat change stream isn't useful enough?  since it doesn't contain the indirect changes?
[18:17:46] <milimetric>	 nah, this wouldn't go above double digit US$millions
[18:18:15] <ottomata>	 i dunno, if dep tracking really happens...we'd likely have a graph db and a way to update it
[18:18:23] <ottomata>	 might fit in well
[18:20:44] <milimetric>	 if we had a category change stream, we could build the category graph on top of any data source (an actual graph db might be best, donno), and then query that to get any level of membership
[18:20:46] <ottomata>	 nuria_:  ok, i brought back my bit about Special:Recentchanges in the blog post and edited a bit
[18:20:54] <ottomata>	 should we send this off to blog people?
[18:21:34] <milimetric>	 (as long as we could populate the category change stream historically, which would involve parsing old revision text)
[18:22:50] <ottomata>	 milimetric:  which we will be able to do by.....december?! :o
[18:24:46] <milimetric>	 yeah, if nothing goes too terribly wrong with that goal.  If we do that and choose a good way to represent the category graph, we should be good to do a lot of things, not just what you're talking about above
[18:42:42] <nuria_>	 ottomata: ya, totally, let's cc melody
[20:36:15] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10EventBus, 10ORES, and 3 others: Fix "score_schema" -- invalid JSON Schema - https://phabricator.wikimedia.org/T197828#4313332 (10Halfak) https://github.com/wiki-ai/revscoring/pull/404
[20:36:23] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10EventBus, 10ORES, and 3 others: Fix "score_schema" -- invalid JSON Schema - https://phabricator.wikimedia.org/T197828#4313333 (10Halfak) a:05Ottomata>03Halfak
[20:54:38] <wikibugs>	 10Analytics, 10Product-Analytics, 10Patch-For-Review, 10SEO: Make various auth libraries available on stat* machines - https://phabricator.wikimedia.org/T197896#4313359 (10Ottomata)
[20:55:57] <wikibugs>	 10Analytics, 10Product-Analytics, 10Patch-For-Review, 10SEO: Make various auth libraries available on stat* machines - https://phabricator.wikimedia.org/T197896#4313361 (10Ottomata) oauth2client and oauthlib were easy because they already have .deb packages in Debian.   We'll have to make a .deb package fo...
[21:10:10] <wikibugs>	 (03Abandoned) 10Ottomata: [WIP] spark streaming playtime [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/201474 (owner: 10Ottomata)
[21:10:21] <wikibugs>	 (03Abandoned) 10Ottomata: [WIP] POC for Realtime Trending Pageviews [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/225485 (owner: 10Ottomata)
[21:14:44] <wikibugs>	 10Analytics-Kanban, 10EventBus, 10Wikimedia-Stream, 10Services (watching), 10User-mobrovac: EventStreams - https://phabricator.wikimedia.org/T130651#4313385 (10Ottomata)
[21:14:48] <wikibugs>	 10Analytics, 10EventBus, 10Wikimedia-Stream, 10Patch-For-Review: Implement server side filtering (if we should) - https://phabricator.wikimedia.org/T152731#4313384 (10Ottomata) 05Open>03declined
[21:28:28] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10EventBus, 10ORES, and 4 others: Modify revision-score schema so that model probabilities won't conflict - https://phabricator.wikimedia.org/T197000#4313449 (10Halfak)
[21:30:16] <wikibugs>	 10Analytics, 10EventBus, 10ORES, 10Patch-For-Review, and 3 others: Invalid field names in ORES models causing downstream Hive ingestion to fail - https://phabricator.wikimedia.org/T195979#4243147 (10Halfak) Well, the field names have nothing wrong with them.  Essentially, "true" and "false" are very useful...
[21:34:38] <wikibugs>	 10Analytics, 10Operations, 10hardware-requests: eqiad: (1) new stat box to offload users from stat1005 - https://phabricator.wikimedia.org/T196345#4253175 (10Tbayer) >>! In T196345#4312713, @elukey wrote: >>>! In T196345#4312349, @RobH wrote: >> We don't have to move users off a box as soon as the warranty e...
[22:13:17] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10User-Elukey: Expand the Hadoop Journal nodes from 3 to 5 to improve resiliency - https://phabricator.wikimedia.org/T189105#4313630 (10Nuria) 05Open>03Resolved
[22:15:42] <wikibugs>	 10Analytics, 10Analytics-Cluster, 10Patch-For-Review: Port Kafka clients to new jumbo cluster - https://phabricator.wikimedia.org/T175461#4313633 (10Nuria)
[22:15:47] <wikibugs>	 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review, 10Services (doing): Move EventStreams to main Kafka clusters - https://phabricator.wikimedia.org/T185225#4313632 (10Nuria) 05Open>03Resolved
[22:16:05] <wikibugs>	 10Analytics-Kanban, 10RESTBase-API, 10Patch-For-Review, 10Services (doing): Analyze surge of traffic in AQS that lead to 504s - https://phabricator.wikimedia.org/T190213#4313637 (10Nuria) 05Open>03Resolved
[23:05:11] <wikibugs>	 10Analytics, 10Operations, 10Traffic: Size of headers processed by varnish? - https://phabricator.wikimedia.org/T198152#4313704 (10Nuria)
[23:09:54] <wikibugs>	 10Analytics-Kanban, 10Patch-For-Review: Fix failing webrequest hours (upload and text 2018-06-14-11) - https://phabricator.wikimedia.org/T197281#4313740 (10Nuria) 05Open>03Resolved
[23:10:07] <wikibugs>	 10Analytics, 10Analytics-Kanban: Access request for Superset: - https://phabricator.wikimedia.org/T196458#4313741 (10Nuria) 05Open>03Resolved
[23:10:30] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Wikimedia-Stream, 10Patch-For-Review: Support timestamp based consumption in KafkaSSE and EventStreams - https://phabricator.wikimedia.org/T196009#4313742 (10Nuria) 05Open>03Resolved
[23:10:47] <wikibugs>	 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review, 10Services (doing): Move EventStreams to main Kafka clusters - https://phabricator.wikimedia.org/T185225#4313745 (10Nuria)
[23:10:50] <wikibugs>	 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review, 10Services (watching): Support connection/rate limiting in EventStreams - https://phabricator.wikimedia.org/T196553#4313744 (10Nuria) 05Open>03Resolved
[23:11:08] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Discovery, 10Discovery-Analysis, 10Product-Analytics: Private data access for non-person user that calculates metrics - https://phabricator.wikimedia.org/T174110#4313748 (10Nuria)
[23:11:12] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review: Puppet admin module should support adding system users to managed groups - https://phabricator.wikimedia.org/T174465#4313747 (10Nuria) 05Open>03Resolved
[23:11:33] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10Services (watching): Enable multiple topics in EventStreams URL - https://phabricator.wikimedia.org/T187418#4313750 (10Nuria) 05Open>03Resolved
[23:11:37] <wikibugs>	 10Analytics, 10Cloud-VPS, 10EventBus, 10Services (watching): Set up a Cloud VPS Kafka Cluster with replicated eventbus production data - https://phabricator.wikimedia.org/T187225#4313751 (10Nuria)
[23:11:50] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Update UA parser - https://phabricator.wikimedia.org/T189230#4313752 (10Nuria) 05Open>03Resolved
[23:12:35] <wikibugs>	 10Analytics-Kanban, 10Patch-For-Review: Use user agent + IP to group anonymous users in geowiki (now geoeditors) - https://phabricator.wikimedia.org/T194170#4313754 (10Nuria) 05Open>03Resolved
[23:12:52] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Wikimedia-Stream, 10Patch-For-Review, 10Wikimedia-Incident: Alerts for common/important EventStreams topic volume - https://phabricator.wikimedia.org/T174493#4313755 (10Nuria) 05Open>03Resolved
[23:13:06] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats 2 Backend: Resiliency, Rollback and Deployment of Data - https://phabricator.wikimedia.org/T177965#4313757 (10Nuria)
[23:13:08] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Make mediawiki-history-reduced table permanent (snapshot partitioning) - https://phabricator.wikimedia.org/T192482#4313756 (10Nuria) 05Open>03Resolved
[23:13:30] <wikibugs>	 10Analytics, 10Analytics-Kanban: Update anonymous grouping to use User Agent - https://phabricator.wikimedia.org/T193415#4313758 (10Nuria) 05Open>03declined
[23:13:47] <wikibugs>	 10Analytics-Kanban: Update oozie druid loading job to facilitate test indexation and prevent prod indexation by mistake - https://phabricator.wikimedia.org/T195882#4313760 (10Nuria) 05Open>03Resolved
[23:14:14] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10Services (watching): Use --new.consumer for main codfw <-> eqiad Kafka MirrorMaker - https://phabricator.wikimedia.org/T190940#4313761 (10Nuria) 05Open>03Resolved
[23:14:34] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10Services (watching): Upgrade Kafka on main cluster with security features - https://phabricator.wikimedia.org/T167039#4313762 (10Nuria) 05Open>03Resolved
[23:15:56] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10Services (watching): EventBus service can drop a few messages during kafka leadership change - https://phabricator.wikimedia.org/T196077#4313763 (10Nuria) 05Open>03Resolved
[23:18:02] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10Services (watching): Enable TLS and authorization for cross DC MirrorMaker - https://phabricator.wikimedia.org/T196081#4313768 (10Nuria) 05Open>03Resolved
[23:18:28] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Discovery, 10EventBus, and 4 others: Increase kafka event retention to 31 - https://phabricator.wikimedia.org/T187296#4313769 (10Nuria) 05Open>03Resolved
[23:18:31] <wikibugs>	 10Analytics, 10Discovery, 10EventBus, 10Wikidata, and 5 others: Create reliable change stream for specific wiki - https://phabricator.wikimedia.org/T161731#4313770 (10Nuria)
[23:23:13] <nuria_>	 stashbot: yt?
[23:23:14] <stashbot>	 See https://wikitech.wikimedia.org/wiki/Tool:Stashbot for help.
[23:23:40] <wikibugs>	 10Analytics, 10Discovery, 10EventBus, 10Wikidata, and 5 others: Create reliable change stream for specific wiki - https://phabricator.wikimedia.org/T161731#4313774 (10Nuria) Ping @Smalyshev now that you have a reliable stream on the new kafka cluster (that supports time-based consumption)  is there any oth...
[23:27:59] <wikibugs>	 10Analytics, 10Discovery, 10EventBus, 10Wikidata, and 5 others: Create reliable change stream for specific wiki - https://phabricator.wikimedia.org/T161731#4313778 (10Smalyshev) @Nuria I don't see any immediate blockers so far.
[23:30:25] <wikibugs>	 10Analytics, 10Discovery, 10EventBus, 10Wikidata, and 5 others: Create reliable change stream for specific wiki - https://phabricator.wikimedia.org/T161731#4313780 (10Ottomata) OO yes @Smalyshev and in case you didn't see, we also increased retention of mediawiki topics to 31 days in the main kafka clusters.
[23:44:42] <wikibugs>	 10Analytics, 10Product-Analytics, 10Patch-For-Review, 10SEO: Make various auth libraries available on stat* machines - https://phabricator.wikimedia.org/T197896#4313799 (10mpopov) >>! In T197896#4313361, @Ottomata wrote: > oauth2client and oauthlib were easy because they already have .deb packages in Debia...