[06:22:35] 10DBA: Investigate possible memory leak on db1115 - https://phabricator.wikimedia.org/T231769 (10Marostegui) >>! In T231769#5643315, @jcrespo wrote: > Based on trending, it will start swapping around 18 december if not restarted. I will restart it on Monday 18th - thanks for rechecking this :) [06:44:17] 10DBA: Remove ar_comment from sanitarium triggers - https://phabricator.wikimedia.org/T234704 (10Marostegui) [06:58:06] 10DBA, 10Operations, 10Patch-For-Review: Decommission db2048.codfw.wmnet - https://phabricator.wikimedia.org/T237913 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by marostegui@cumin1001 for hosts: `db2048.codfw.wmnet` - db2048.codfw.wmnet (**PASS**) - Downtimed host on Icinga - Downt... [07:00:36] 10DBA, 10Operations: Decommission db2043-db2069 - https://phabricator.wikimedia.org/T228258 (10Marostegui) [07:25:49] 10DBA, 10Operations: Switchover s1 primary database master db1067 -> db1083 - 14th Nov 05:00 - 05:30 UTC - https://phabricator.wikimedia.org/T234800 (10Marostegui) I just realised that when we created this task and scheduled the maintenance I didn't take into account that we'd have had the change from CEST to... [07:26:36] 10DBA, 10Operations: Switchover s1 primary database master db1067 -> db1083 - 14th Nov 06:00 - 06:30 UTC - https://phabricator.wikimedia.org/T234800 (10Marostegui) [07:26:49] 10DBA: Switchover s6 primary database master db1061 -> db1131 - 19th Nov 05:00 - 05:30 UTC - https://phabricator.wikimedia.org/T235469 (10Marostegui) I just realised that when we created this task and scheduled the maintenance I didn't take into account that we'd have had the change from CEST to CET, so I am goi... [07:27:29] 10DBA: Switchover s6 primary database master db1061 -> db1131 - 19th Nov 06:00 - 06:30 UTC - https://phabricator.wikimedia.org/T235469 (10Marostegui) [07:37:47] 10DBA: Drop labtestwikitech database from m5 - https://phabricator.wikimedia.org/T236010 (10Marostegui) [09:22:52] 10DBA, 10Operations, 10serviceops, 10Goal, 10Patch-For-Review: Switchover backup director service from helium to backup1001 - https://phabricator.wikimedia.org/T236406 (10jcrespo) 05Open→03Resolved I am going to consider the switchover as done, and create a separate task for followups. [09:22:56] 10DBA, 10Operations, 10serviceops, 10Goal, 10Patch-For-Review: Strengthen backup infrastructure and support - https://phabricator.wikimedia.org/T229209 (10jcrespo) [09:25:43] 10DBA, 10Operations: Switchover s7 primary database master db1062 -> db1086 - 26th Nov 06:00 - 06:30 UTC - https://phabricator.wikimedia.org/T238044 (10Marostegui) [09:25:59] 10DBA, 10Operations: Decommission db1061-db1073 - https://phabricator.wikimedia.org/T217396 (10Marostegui) [09:26:01] 10DBA, 10Operations: Switchover s7 primary database master db1062 -> db1086 - 26th Nov 06:00 - 06:30 UTC - https://phabricator.wikimedia.org/T238044 (10Marostegui) [09:26:04] 10DBA: Failover DB masters in row D - https://phabricator.wikimedia.org/T186188 (10Marostegui) [09:27:15] 10DBA, 10Operations: Switchover s7 primary database master db1062 -> db1086 - 26th Nov 06:00 - 06:30 UTC - https://phabricator.wikimedia.org/T238044 (10Marostegui) p:05Triage→03Normal [09:31:11] 10DBA, 10Operations: Switchover s7 primary database master db1062 -> db1086 - 26th Nov 06:00 - 06:30 UTC - https://phabricator.wikimedia.org/T238044 (10Marostegui) [09:31:24] 10DBA, 10Operations, 10User-notice: Switchover s7 primary database master db1062 -> db1086 - 26th Nov 06:00 - 06:30 UTC - https://phabricator.wikimedia.org/T238044 (10Marostegui) [09:35:57] 10DBA, 10Operations, 10serviceops, 10Goal, 10Patch-For-Review: Strengthen backup infrastructure and support - https://phabricator.wikimedia.org/T229209 (10jcrespo) [09:44:51] 10DBA: Drop labtestwikitech database from m5 - https://phabricator.wikimedia.org/T236010 (10Marostegui) >>! In T236010#5590224, @Marostegui wrote: > These are the active grants on the DB: > ` > GRANT ALL PRIVILEGES ON `labtestwiki`.* TO 'wikiadmin'@'10.192.32.22' > GRANT ALL PRIVILEGES ON `labtestwiki`.* TO 'wik... [09:45:06] 10DBA: Drop labtestwikitech database from m5 - https://phabricator.wikimedia.org/T236010 (10Marostegui) [09:47:33] 10DBA: Drop labtestwikitech database from m5 - https://phabricator.wikimedia.org/T236010 (10Marostegui) [09:47:59] 10DBA: Drop labtestwikitech database from m5 - https://phabricator.wikimedia.org/T236010 (10Marostegui) [09:49:15] /win 28 [10:32:15] 10Blocked-on-schema-change, 10DBA: Schema change to rename user_newtalk indexes - https://phabricator.wikimedia.org/T234066 (10Marostegui) [10:32:26] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) [10:34:12] 10DBA: Drop labtestwikitech database from m5 - https://phabricator.wikimedia.org/T236010 (10Marostegui) I have finally dropped the `labtestwiki` database from m5, it has had its tabled renamed for a long time without any issues: T233236#5585652 Backup is at: `db1133:/root/T236010_backup_labtestwiki_.sql` [10:34:19] 10DBA: Drop labtestwikitech database from m5 - https://phabricator.wikimedia.org/T236010 (10Marostegui) 05Open→03Resolved [11:08:16] 10DBA: Compress new Wikibase tables - https://phabricator.wikimedia.org/T232446 (10Marostegui) [11:14:28] 10DBA, 10Operations: Predictive failures on disk S.M.A.R.T. status - https://phabricator.wikimedia.org/T208323 (10Marostegui) [11:17:24] 10DBA: Compress new Wikibase tables - https://phabricator.wikimedia.org/T232446 (10Ladsgroup) Why this is created in s4? Does anyone knows why? [11:27:05] 10DBA: Compress new Wikibase tables - https://phabricator.wikimedia.org/T232446 (10Marostegui) >>! In T232446#5655628, @Ladsgroup wrote: > Why this is created in s4? Does anyone knows why? That was part of the Commons Structured Data project. We advised against doing this, and rather, creating a new section (wi... [11:35:21] 10Blocked-on-schema-change, 10DBA: Schema change to rename user_newtalk indexes - https://phabricator.wikimedia.org/T234066 (10Marostegui) s5 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1003 [] db1130 [] db1124 [] db1113 [] db1110 [] db1102 [] db1100 [] db1097 [] db1096 []... [11:35:24] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) s5 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1003 [] db1130 [] db1124 [] db1113 [] db1110 [] db110... [12:54:02] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for mnwwiki - https://phabricator.wikimedia.org/T235743 (10Bstorm) @Marostegui if you can get those steps, that would be great. I'm at tech conf, and it might be simpler for whoever grabs this. [12:59:27] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for mnwwiki - https://phabricator.wikimedia.org/T235743 (10Marostegui) >>! In T235743#5655940, @Bstorm wrote: > @Marostegui if you can get those steps, that would be great. I'm at tech conf, and it might be simpler for wh... [17:02:49] 10DBA, 10Data-Services, 10Operations, 10cloud-services-team (Kanban): Prepare and check storage layer for gcrwiki - https://phabricator.wikimedia.org/T238114 (10Marostegui) Please let us know when the database is created so we can sanitize it on labs, before sending it to the cloud team for the views creation [17:03:08] 10DBA, 10Data-Services, 10Operations, 10cloud-services-team (Kanban): Prepare and check storage layer for shywiktionary - https://phabricator.wikimedia.org/T238115 (10Marostegui) Please let us know when the database is created so we can sanitize it on labs, before sending it to the cloud team for the views... [17:43:50] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: (codfw):rack/setup/install db213[2-5] - https://phabricator.wikimedia.org/T237702 (10Papaul) [18:37:12] 10DBA, 10Analytics: Repurpose db1107 as a generic database - https://phabricator.wikimedia.org/T238113 (10elukey) [19:28:02] 10DBA, 10MediaWiki-Logging, 10Core Platform Team Workboards (Clinic Duty Team), 10Performance Issue, 10Wikimedia-production-error: Page creation log cannot be viewed from oldest records, Fatal: "execution time limit of 60 seconds was exceeded" - https://phabricator.wikimedia.org/T237026 (10Anomie) The q... [22:32:03] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: (codfw):rack/setup/install db213[2-5] - https://phabricator.wikimedia.org/T237702 (10Papaul) ` papaul@asw-a-codfw# show | compare [edit interfaces interface-range vlan-private1-a-codfw] member xe-7/0/9 { ... } + member ge-5/0/1; [edit interface... [22:35:57] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: (codfw):rack/setup/install db213[2-5] - https://phabricator.wikimedia.org/T237702 (10Papaul) [23:32:42] 10DBA, 10MediaWiki-Comment-backend, 10Patch-For-Review, 10Schema-change: Fix WMF schemas to not break when comment store goes WRITE_NEW - https://phabricator.wikimedia.org/T187089 (10DannyS712)