[05:04:54] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review, 10User-notice: Switchover m1 primary master: db1063 to db1135: Tuesday 10th September at 16:00 UTC - https://phabricator.wikimedia.org/T231403 (10Marostegui) I have reserved the window on the Deployments page.
[05:05:03] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review: Switchover s8 (wikidata) primary database master db1104 -> db1109 - 10th Sept @05:00 UTC - https://phabricator.wikimedia.org/T230762 (10Marostegui) I have reserved the window on the Deployments page.
[05:09:09] <wikibugs>	 10DBA, 10Operations: Decommission db2046.codfw.wmnet - https://phabricator.wikimedia.org/T231767 (10Marostegui)
[05:18:05] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review: Decommission db2046.codfw.wmnet - https://phabricator.wikimedia.org/T231767 (10Marostegui)
[05:37:10] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review, 10User-notice: Switchover m1 primary master: db1063 to db1135: Tuesday 10th September at 16:00 UTC - https://phabricator.wikimedia.org/T231403 (10Marostegui)
[06:32:20] <wikibugs>	 10DBA, 10CPT Initiatives (Core REST API in PHP), 10Core Platform Team Workboards (Green): Compose Count Queries - https://phabricator.wikimedia.org/T231598 (10jcrespo) Question- this use case seems very similar to category counts. Wouldn't be more useful to keep this "cached" on counters and increment on edi...
[06:43:55] <jynus>	 I am guessing you recently manually run s6 snapshot prepare?
[07:11:49] <marostegui>	 nope
[07:11:51] <marostegui>	 I haven't
[07:13:44] <jynus>	 strange, because there is a recently complated backup 2713 | snapshot.s6.2019-08-30--02-33-45
[07:14:07] <jynus>	 but whose preparation started at 2019-09-06 03:29:57
[07:14:42] <marostegui>	 Not sure what that would be, but definitely not me :)
[07:15:01] <jynus>	 yeah, something strange
[07:49:42] <wikibugs>	 10DBA, 10CPT Initiatives (Core REST API in PHP), 10Core Platform Team Workboards (Green): Compose New History Queries - https://phabricator.wikimedia.org/T231599 (10Marostegui) I have been reviewing @Anomie's queries and I haven't found anything strange there. I have checked those queries (with Obama's rev_p...
[08:35:09] <wikibugs>	 10DBA, 10Operations: Decommission db2043-db2069 - https://phabricator.wikimedia.org/T228258 (10Marostegui)
[09:17:37] <jynus>	 I am taking care of s3 on both dcs
[09:17:41] <jynus>	 *s3 backups
[09:17:50] <marostegui>	 thanks
[09:18:01] <jynus>	 I am considering stopping replication on all servers
[09:18:12] <marostegui>	 stop, backup, start?
[09:18:18] <jynus>	 yeah
[09:18:21] <jynus>	 it is just an option
[09:18:40] <marostegui>	 yeah, we discussed that in the past, and I am fine
[09:35:32] <wikibugs>	 10DBA, 10conftool: #dbctl: add 'comment'/'description' metadata to instances - https://phabricator.wikimedia.org/T229677 (10Marostegui) >>! In T229677#5412237, @gerritbot wrote: > Change 529396 **merged** by jenkins-bot: > [operations/software/conftool@master] dbctl: add note & candidate_master fields >  > [[h...
[10:05:06] <wikibugs>	 10DBA, 10conftool: specify group (api/vslow/etc) weights in terms of 0..100 instead of 0..1 - https://phabricator.wikimedia.org/T231018 (10Marostegui) p:05Triage→03Normal
[10:12:22] <wikibugs>	 10DBA: Investigate possible memory leak on db1115 - https://phabricator.wikimedia.org/T231769 (10Marostegui) It has grown 50GB in 2 days. Right now we are at 90GB, and it increases around 2GB every 8 hours. TokuDB buffer + Innodb buffer are 50GB, nowhere near those 90GB (and still increasing).  On Monday I am go...
[10:54:26] <Amir1>	 marostegui: jynus I keep this updated on status of the work: https://phabricator.wikimedia.org/T232194
[11:51:21] <marostegui>	 thanks Amir1 
[11:51:24] <marostegui>	 I subscribed myself
[12:12:54] <jynus>	 "[ERROR]: Mediawiki section to failover: es0, sections on the master: No sections, sections on the replica: No sections"
[12:13:20] <marostegui>	 That's supposed to happen, as es isn't supported yet
[12:13:34] <jynus>	 also there is no es0 section :-D
[12:14:06] <jynus>	 I am doing --mediawiki-section=es0 to make sure validations work
[12:14:20] <marostegui>	 nice!
[12:17:45] <jynus>	 "[ERROR]: es2001.codfw.wmnet is not the configured master of the es0 section in any datacenter according to mediawiki: provide the right section or skip this check by not providing a mediawiki section at all"
[12:18:24] <marostegui>	 cooool! :)
[12:26:40] <wikibugs>	 10DBA, 10conftool: #dbctl: add 'comment'/'description' metadata to instances - https://phabricator.wikimedia.org/T229677 (10CDanis) >>! In T229677#5470748, @Marostegui wrote: > I saw this was merged - thanks! > From that commit message there is still some work to be done, do you have a rough ETA on when we'll...
[12:27:56] <wikibugs>	 10DBA, 10conftool: #dbctl: add 'comment'/'description' metadata to instances - https://phabricator.wikimedia.org/T229677 (10Marostegui) >>! In T229677#5471181, @CDanis wrote: >  > I had been thinking I'd release the changes for this and also {T229686} at the same time, but I won't be done with that before goin...
[16:24:32] <wikibugs>	 10DBA, 10CPT Initiatives (Core REST API in PHP), 10Core Platform Team Workboards (Green): Compose New History Queries - https://phabricator.wikimedia.org/T231599 (10Anomie) >>! In T231599#5470553, @Marostegui wrote: > However, I am seeing different query plans (with less scanned rows) than the ones Brad saw:...
[16:43:59] <wikibugs>	 10DBA, 10CPT Initiatives (Core REST API in PHP), 10Core Platform Team Workboards (Green): Compose Count Queries - https://phabricator.wikimedia.org/T231598 (10Anomie) Here are corrected versions of the bot and anon queries.  > [ ] botedits: total number of edits by bots >   [ ] Where a bot is defined as a Us...