[03:09:47] <wikibugs>	 DBA, Operations: m1-master switch from db1001 to db1016 - https://phabricator.wikimedia.org/T106312#2395548 (Dzahn) I can check on RT ('rt') and racktables (the other rt) on Wednesday, maybe around 18UTC but i dont worry about it since these services are just used by ops themselves.  That said, please no...
[07:20:45] <wikibugs>	 DBA, Operations: m1-master switch from db1001 to db1016 - https://phabricator.wikimedia.org/T106312#2395761 (akosiaris) >>! In T106312#2394393, @jcrespo wrote: > @Akosiaris @MoritzMuehlenhoff @fgiunchedi @demon @Krenair @Dzahn I intend to perform the failover on Wednesday 22, 16:00 UTC.  >  > I do not re...
[08:46:47] <wikibugs>	 DBA, Operations: m1-master switch from db1001 to db1016 - https://phabricator.wikimedia.org/T106312#2395891 (fgiunchedi) >>! In T106312#2395761, @akosiaris wrote: >>>! In T106312#2394393, @jcrespo wrote: >> @Akosiaris @MoritzMuehlenhoff @fgiunchedi @demon @Krenair @Dzahn I intend to perform the failover...
[08:47:33] <wikibugs>	 DBA, Operations: m1-master switch from db1001 to db1016 - https://phabricator.wikimedia.org/T106312#2395893 (MoritzMuehlenhoff) 17 UTC also fine with me
[09:00:01] <wikibugs>	 DBA, Operations: m1-master switch from db1001 to db1016 - https://phabricator.wikimedia.org/T106312#2395900 (jcrespo) Let's go with [[ https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20160622T1500 | 17 UTC ]] instead
[09:06:17] <wikibugs>	 DBA, Labs, Tool-Labs: Some users use persistent connections that are idle, wasting memory and other resources that could be used for other users - https://phabricator.wikimedia.org/T138283#2395904 (jcrespo)
[09:09:31] <wikibugs>	 DBA, Labs, Tool-Labs: Some users use persistent connections that are idle, wasting memory and other resources that could be used for other users - https://phabricator.wikimedia.org/T138283#2395919 (jcrespo) @PeterBowman https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Database#Connection_handling_p...
[09:51:30] <wikibugs>	 DBA, Operations: Decomission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#2395956 (jcrespo) a:jcrespo
[09:52:33] <wikibugs>	 DBA: Rampant differences in indexes on enwiki.revision across the DB cluster - https://phabricator.wikimedia.org/T132416#2395961 (jcrespo)
[09:52:33] <wikibugs>	 DBA: Create a whitelist of tables to checksum on all wikis - https://phabricator.wikimedia.org/T104310#2395962 (jcrespo)
[09:52:36] <wikibugs>	 DBA, Operations: Upgrade db1022, which has an older kernel - https://phabricator.wikimedia.org/T101516#2395963 (jcrespo)
[09:52:38] <wikibugs>	 DBA: Automatize the check and fix of object, schema and data drifts between production masters and slaves - https://phabricator.wikimedia.org/T104459#2395958 (jcrespo) stalled>Open a:jcrespo
[10:34:17] <wikibugs>	 DBA, Datasets-General-or-Unknown: Select of revisions for stub history files does not explicitly order revisions - https://phabricator.wikimedia.org/T29112#2396016 (ArielGlenn) I've done a bunch of timing tests and explains (still shaking my head at the query optimizer) but still more to go; these were o...
[10:39:48] <wikibugs>	 DBA, Datasets-General-or-Unknown: Select of revisions for stub history files does not explicitly order revisions - https://phabricator.wikimedia.org/T29112#2396023 (jcrespo) Are those test positive or negative?
[11:40:22] <wikibugs>	 DBA, Labs, Tool-Labs: Some users use persistent connections that are idle, wasting memory and other resources that could be used for other users - https://phabricator.wikimedia.org/T138283#2396069 (PeterBowman) I'm sorry, this shouldn't have happened. I've killed those processes and disabled any scri...
[11:49:28] <wikibugs>	 DBA, Discovery, Labs, Maps, Tool-Labs: Some users use persistent connections that are idle, wasting memory and other resources that could be used for other users - https://phabricator.wikimedia.org/T138283#2396104 (valhallasw) 10.68.16.70 is maps-wma1.maps.eqiad.wmflabs.
[13:31:24] <wikibugs>	 DBA, Labs, Tool-Labs, Tool-Labs-tools-Database-Queries: Run a Tool Labs query without Timing out - https://phabricator.wikimedia.org/T138111#2396303 (chasemp) p:Triage>Normal a:yuvipanda>None
[13:33:28] <wikibugs>	 DBA, Labs, Tool-Labs, Tool-Labs-tools-Database-Queries: Run a Tool Labs query without Timing out - https://phabricator.wikimedia.org/T138111#2396315 (jcrespo) I was wrong, you are executing 1000 times a 32-million row query (but the conclusions still stand).
[13:39:10] <wikibugs>	 DBA, Labs: Wrong page title in labs database replica enwiki page table - https://phabricator.wikimedia.org/T136618#2396380 (chasemp) p:Triage>High
[13:43:00] <wikibugs>	 DBA, Labs: Missing data on labs replica database - https://phabricator.wikimedia.org/T133715#2396434 (chasemp) p:Triage>Normal
[13:47:14] <wikibugs>	 DBA, Labs: Discrepancy between labsdb replicas of arwiki_p.user_groups - https://phabricator.wikimedia.org/T133469#2396475 (chasemp) p:Triage>Normal
[13:49:05] <wikibugs>	 DBA, Labs, Labs-Infrastructure, Operations: Migrate labsdb1005/1006/1007 to jessie - https://phabricator.wikimedia.org/T123731#2396500 (chasemp) p:Triage>Normal one thought is we have an influx of new labsdb things coming I believe. This way sort itself out w/o a lot of in-place shuffling.
[18:28:57] <wikibugs>	 DBA, Datasets-General-or-Unknown: Select of revisions for stub history files does not explicitly order revisions - https://phabricator.wikimedia.org/T29112#2397374 (ArielGlenn) They mean that I need to be very careful about page ranges; it's a sort of sliding scale.  I don't have to worry about that in t...
[18:35:29] <wikibugs>	 DBA, Labs, Labs-Infrastructure, Operations, Blocked-on-Operations: adywiki is missing the associated adywiki_p database with appropriate views - https://phabricator.wikimedia.org/T135029#2286195 (Gehel)
[18:56:13] <wikibugs>	 DBA, Labs, Labs-Infrastructure, Operations, Blocked-on-Operations: adywiki and jamwiki are missing the associated *_p databases with appropriate views - https://phabricator.wikimedia.org/T135029#2397393 (MaxSem)
[21:58:48] <wikibugs>	 DBA, Labs, Labs-Infrastructure, Operations, Blocked-on-Operations: adywiki and jamwiki are missing the associated *_p databases with appropriate views - https://phabricator.wikimedia.org/T135029#2397893 (Gehel) Summary of a discussion with @ori:  The maintain-replicas script creates a new sch...
[22:37:56] <wikibugs>	 DBA, Labs, Labs-Infrastructure, Operations, Blocked-on-Operations: adywiki and jamwiki are missing the associated *_p databases with appropriate views - https://phabricator.wikimedia.org/T135029#2397950 (Krenair) I agree, although it needs to be a separate ticket and I don't think we can just...
[22:39:02] <wikibugs>	 Blocked-on-schema-change, DBA, Notifications, Collab-Team-2016-Apr-Jun-Q4, Schema-change: Schema changes for Echo moderation - https://phabricator.wikimedia.org/T135699#2397955 (Catrope) >>! In T135699#2334730, @Catrope wrote: >> I've also found that there are lots of orphaned rows in `echo_e...