[05:28:59] 10DBA, 10Wikimedia-Site-requests, 07Performance: Disable miser mode ($wgMiserMode) on small wikis (wikis in small.dblist) - https://phabricator.wikimedia.org/T48098#3020665 (10Dereckson) @jcrespo Do you have some thoughts on this topic? I'd like to create a list of ten s3 wikis, one latin, one non latin in e... [05:37:55] 10DBA, 10Wikimedia-Site-requests, 13Patch-For-Review: Recreate a wiki for Wikimedia Portugal - https://phabricator.wikimedia.org/T126832#3020674 (10Dereckson) >>! In T126832#3016561, @jcrespo wrote: > This worries me: T157636#3012112 > > @Dereckson: let's schedule the deletion and the recreation at the same... [07:08:04] 10DBA, 06Labs, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3020807 (10Marostegui) commonswiki has been compressed on db1095 and labsdb1009. It has gone from around 1.3T to 495G. There is still 1.6T available on db1... [07:11:50] 10DBA: s5: db1070 not using file per table - https://phabricator.wikimedia.org/T157931#3020808 (10Marostegui) [09:44:14] 07Blocked-on-schema-change, 06Collaboration-Team-Triage, 10Notifications, 13Patch-For-Review, 07Schema-change: Add primary key to echo_notification table - https://phabricator.wikimedia.org/T136428#3021146 (10Marostegui) db1077: ``` root@neodymium:~# for i in mediawikiwiki officewiki ; do echo $i; mysql... [09:46:47] 07Blocked-on-schema-change, 06Collaboration-Team-Triage, 10Notifications, 13Patch-For-Review, 07Schema-change: Add primary key to echo_notification table - https://phabricator.wikimedia.org/T136428#3021153 (10Marostegui) s3 is done for officewiki and mediawikiwiki So, s3 and s7 are done. Pending: x1 ```... [09:52:08] 10DBA, 13Patch-For-Review: Fix dbstore2001 and dbstore2002 - https://phabricator.wikimedia.org/T130128#3021163 (10Marostegui) >>! In T130128#3017721, @jcrespo wrote: > I have enabled delayed replicationon dbstore2001- it is not dangerous, as worse case scenario it just stops the replication thread more than it... [09:58:41] 10DBA, 13Patch-For-Review: Fix dbstore2001 and dbstore2002 - https://phabricator.wikimedia.org/T130128#3021169 (10Marostegui) >>! In T130128#3021163, @Marostegui wrote: >>>! In T130128#3017721, @jcrespo wrote: >> I have enabled delayed replicationon dbstore2001- it is not dangerous, as worse case scenario it j... [09:58:44] 10DBA, 10Wikimedia-Site-requests, 07Performance: Disable miser mode ($wgMiserMode) on small wikis (wikis in small.dblist) - https://phabricator.wikimedia.org/T48098#3021170 (10jcrespo) Yes, keep me in the loop. [10:02:21] 10DBA, 10Wikimedia-Site-requests, 13Patch-For-Review: Recreate a wiki for Wikimedia Portugal - https://phabricator.wikimedia.org/T126832#3021172 (10jcrespo) Let's aim for the 23, if that is ok for you, but with the possibility of delaying it if we need more time for preparation/something goes wrong before th... [10:16:01] 10DBA, 13Patch-For-Review: Fix dbstore2001 and dbstore2002 - https://phabricator.wikimedia.org/T130128#3021210 (10jcrespo) @Marostegui You can look at the delay easier* by using the heartbeat table: ```lang=sql $ mysql -h dbstore2001.codfw.wmnet heartbeat -e "SELECT shard, greatest(0, TIMESTAMPDIFF(MICROSECO... [13:59:23] 10DBA, 10Wikimedia-Site-requests, 07Performance: Disable miser mode ($wgMiserMode) on small wikis (wikis in small.dblist) - https://phabricator.wikimedia.org/T48098#3021771 (10Reedy) [13:59:27] 10DBA, 06Operations, 10ops-eqiad: Degraded RAID on db1053 - https://phabricator.wikimedia.org/T151465#3021773 (10Marostegui) @Cmjohnson as per your update on T154587, let me know when you replace db1053's disk. Thanks! [14:00:27] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: db2060 not accessible - https://phabricator.wikimedia.org/T156161#3021775 (10Marostegui) @Papaul db2060 is now off. Once you are done, please power it on and I will take care from there. Thanks! [14:01:08] 10DBA, 06Operations, 10ops-eqiad: Degraded RAID on db1053 - https://phabricator.wikimedia.org/T151465#3021776 (10Cmjohnson) p:05Normal>03Triage Disks arrived....moving this up to do today. [14:01:26] yay [14:01:41] :) [14:02:40] For the hw etherpad where we have the db-eqiad.php, I am going to copy the shards array twice, so we can see the "before" and "after" [14:06:06] mm [14:06:19] maybe then it is better to do an RC? [14:06:24] so we can see the diff? [14:06:52] mmm, don't know, check how it looks now [14:07:11] ah, ok, it is clear [14:07:43] let's get rid of the header comment and let's add x1 [14:07:48] if you are ok [14:07:49] ok [14:33:41] 10DBA: Install and reimage dbstore1001 as jessie - https://phabricator.wikimedia.org/T153768#3021894 (10jcrespo) I have agreed 23 of Feb for the replacement with Chris- this week we have the labsdb failover on the 15, plus it is holidays on Monday on the US and many people will be away. That means we have until... [15:21:35] 10DBA: Install and reimage dbstore1001 as jessie - https://phabricator.wikimedia.org/T153768#3021979 (10Marostegui) >>! In T153768#3021894, @jcrespo wrote: > I have agreed 23 of Feb for the replacement with Chris- this week we have the labsdb failover on the 15, plus it is holidays on Monday on the US and many p... [17:17:52] 10DBA, 10Wikimedia-Site-requests, 07Performance: Disable miser mode ($wgMiserMode) on small wikis (wikis in small.dblist) - https://phabricator.wikimedia.org/T48098#3022300 (10He7d3r) [17:47:35] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: db2060 not accessible - https://phabricator.wikimedia.org/T156161#3022399 (10Papaul) a:05Papaul>03Marostegui Main board replacement complete. [17:59:06] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: db2060 not accessible - https://phabricator.wikimedia.org/T156161#3022468 (10Marostegui) Thanks! I will take it from here! [18:01:18] 10DBA, 10Wikimedia-Site-requests, 13Patch-For-Review: Recreate a wiki for Wikimedia Portugal - https://phabricator.wikimedia.org/T126832#3022477 (10Dereckson) Ok, works for me. [18:02:47] 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 10MediaWiki-extensions-PageAssessments: Update page_assessments_projects schema for subprojects in production - https://phabricator.wikimedia.org/T156305#3022479 (10kaldari) @jcrespo: What's the procedure for getting the table updated on Tool Labs? [18:23:46] 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 10MediaWiki-extensions-PageAssessments: Update page_assessments_projects schema for subprojects in production - https://phabricator.wikimedia.org/T156305#3022516 (10jcrespo) @kaldari, can you check the new labsdb servers, it may be already there? We may... [18:37:39] 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 10MediaWiki-extensions-PageAssessments: Update page_assessments_projects schema for subprojects in production - https://phabricator.wikimedia.org/T156305#3022589 (10kaldari) @jcrespo: The schema has been updated on labsdb1001: ``` mysql:wikiadmin@labsdb1... [20:09:40] 10DBA, 10Wikidata, 07Performance, 15User-Daniel, and 2 others: Build an environment to test change dispatching using Redis-based locking - https://phabricator.wikimedia.org/T155190#3023065 (10Ladsgroup) I think we are done with instance-based tests and we need to move on to beta cluster. I tests I've made...