[00:47:32] <wikibugs>	 10DBA, 10Labs, 10Epic: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3366131 (10MZMcBride)
[00:50:40] <wikibugs>	 10DBA, 10Labs, 10Epic: Labs database replica drift - https://phabricator.wikimedia.org/T138967#2415416 (10MZMcBride) >>! In T138967#3333050, @bd808 wrote: > Cross-post from {T166091}: >  > ```SELECT pl_namespace, pl_title > FROM page > JOIN pagelinks ON pl_from = page_id > WHERE page_namespace=0 AND page_tit...
[04:50:29] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review: Prepare mysql hosts for stretch - https://phabricator.wikimedia.org/T168356#3366331 (10Dzahn)
[05:37:36] <wikibugs>	 10DBA, 10Labs, 10User-bd808, 10cloud-services-team (Kanban): setup dewiki and wikidatawiki on the labsdb1009, 1010 and 1011 - https://phabricator.wikimedia.org/T168021#3366391 (10Marostegui) Excellent! Thanks for letting us know and thanks for testing the new labs servers, please let us know if you see wei...
[05:45:41] <wikibugs>	 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s2 - https://phabricator.wikimedia.org/T166205#3366395 (10Marostegui)
[05:49:15] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: db1016 m1 master: Possibly faulty BBU - https://phabricator.wikimedia.org/T166344#3366397 (10Marostegui) After issuing the relearn, the raid is back to WB: ``` ˜/icinga-wm 7:48> RECOVERY - MegaRAID on db1016 is OK: OK: optimal, 1 logical, 2 physical, WriteBack policy  ```
[05:50:41] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s5 - https://phabricator.wikimedia.org/T166207#3366400 (10Marostegui)
[05:52:54] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s5 - https://phabricator.wikimedia.org/T166207#3366401 (10Marostegui) db1049 is done: ``` root@neodymium:/home/marostegui# for i in `cat s5_tables`; do echo $i; mysql --skip-ssl -hdb1049 dewik...
[06:11:40] <wikibugs>	 10DBA: dbstore2001 s5 thread is 6 days delayed - https://phabricator.wikimedia.org/T168354#3366428 (10Marostegui) During the night, until now, s5 running by itself not only didn't catch up but got further delayed. There is something wrong with this host (or at least its data) - for now and until we get to work o...
[06:22:16] <wikibugs>	 10DBA, 10Analytics: Purge all old data from master - https://phabricator.wikimedia.org/T168414#3366436 (10elukey) eventlogging_cleaner.py should be capable of purging db1046 (simply not doing any whitelisting/update-with-nulls) but we can have another script if simpler.
[07:47:28] <wikibugs>	 10DBA, 10Labs: enwiki_p logging vs logging_userindex returning dramatically different results - https://phabricator.wikimedia.org/T168349#3366564 (10Marostegui) I tried the original two queries on labsdb-web.eqiad.wmnet without the LIMIT. The first query; ``` 12471 rows in set (10 min 52.70 sec) ```  The secon...
[08:21:17] <wikibugs>	 10DBA, 10Patch-For-Review: dbstore2001 s5 thread is 6 days delayed - https://phabricator.wikimedia.org/T168354#3366635 (10Marostegui) a:03Marostegui
[08:23:09] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s5 - https://phabricator.wikimedia.org/T166207#3366637 (10Marostegui) db1082 is done: ``` root@neodymium:/home/marostegui# for i in `cat s5_tables`; do echo $i; mysql --skip-ssl -hdb1082 dewik...
[08:23:34] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s5 - https://phabricator.wikimedia.org/T166207#3366638 (10Marostegui)
[08:36:27] <wikibugs>	 10DBA, 10Patch-For-Review: dbstore2001: s5 thread isn't able to catch up with the master - https://phabricator.wikimedia.org/T168354#3366653 (10Marostegui)
[09:48:13] <wikibugs>	 10DBA, 10Operations: Drop wikilove_image_log table from Wikimedia wikis - https://phabricator.wikimedia.org/T127219#3366760 (10Marostegui) a:03Marostegui
[09:59:40] <wikibugs>	 10DBA, 10Labs, 10Tool-Labs: labsdb1001 and labsdb1003 short on available space - https://phabricator.wikimedia.org/T132431#3366819 (10Marostegui)
[09:59:46] <wikibugs>	 10DBA, 10Labs, 10Tool-Labs: s51127__dewiki_lists (merlbot) database using 13G on labsdb1001 (enwiki) - https://phabricator.wikimedia.org/T133325#3366816 (10Marostegui) 05Open>03Resolved a:03Marostegui And I have dropped this: ``` mysql:root@localhost [s51127__dewiki_lists]> drop database if exists s511...
[10:32:53] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s5 - https://phabricator.wikimedia.org/T166207#3366869 (10Marostegui)
[11:49:15] <wikibugs>	 10Blocked-on-schema-change, 10Collaboration-Team-Triage, 10MediaWiki-extensions-PageCuration, 10Schema-change: Drop ptrl_comment in production - https://phabricator.wikimedia.org/T157762#3367051 (10Marostegui) p:05Triage>03Normal
[12:55:16] <wikibugs>	 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s2 - https://phabricator.wikimedia.org/T166205#3367171 (10Marostegui)
[12:55:31] <wikibugs>	 10Blocked-on-schema-change, 10DBA: Convert unique keys into primary keys for some wiki tables on s1, s2, s4, s5 and s7 (eqiad) - https://phabricator.wikimedia.org/T164185#3367174 (10Marostegui)
[12:55:33] <wikibugs>	 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s2 - https://phabricator.wikimedia.org/T166205#3288300 (10Marostegui) 05Open>03Resolved Everything is now done.
[12:58:52] <wikibugs>	 10Blocked-on-schema-change, 10DBA: Convert unique keys into primary keys for some wiki tables on s7 - https://phabricator.wikimedia.org/T166208#3367191 (10Marostegui)
[13:03:47] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s5 - https://phabricator.wikimedia.org/T166207#3367208 (10Marostegui) db1070 is done: ``` root@neodymium:/home/marostegui# for i in `cat s5_tables`; do echo $i; mysql --skip-ssl -hdb1070 dewik...
[13:03:57] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s5 - https://phabricator.wikimedia.org/T166207#3367209 (10Marostegui)
[13:15:52] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s5 - https://phabricator.wikimedia.org/T166207#3367238 (10Marostegui) #cloud-services-team I am going to deploy this alter table on labsdb1001 - so the s5 shard will get delayed there for arou...