[05:35:13] 10DBA, 10ContentTranslation, 10Data-Services: Consider adding ContentTranslation (CX) tables to wiki replicas - https://phabricator.wikimedia.org/T196020#4245011 (10Marostegui) There was a great job done by @Bawolff who did a security review already (T103011) and for the cx tables it is not completely clear... [07:19:31] 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy - https://phabricator.wikimedia.org/T190704#4245086 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` ['db2... [08:03:03] 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy - https://phabricator.wikimedia.org/T190704#4245167 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` ['db2... [08:03:25] 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy - https://phabricator.wikimedia.org/T190704#4245168 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db2092.codfw.wmnet'] ``` Of which those **FAILED**: ```... [08:03:33] 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy - https://phabricator.wikimedia.org/T190704#4245169 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` ['db2... [08:14:15] 10DBA, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10Patch-For-Review, 10Schema-change: Schema change to make archive.ar_rev_id NOT NULL - https://phabricator.wikimedia.org/T191316#4245175 (10Marostegui) [08:17:40] 10DBA, 10Multi-Content-Revisions, 10Patch-For-Review, 10Schema-change: Schema change to drop archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T192926#4245184 (10Marostegui) [08:19:19] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Make several mediawiki table fields unsigned ints on wmf databases - https://phabricator.wikimedia.org/T89737#4245186 (10Marostegui) [08:19:36] 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Patch-For-Review, 10Wikidata-Ministry-Of-Magic: Schema change for ct_tag_id field to change_tag - https://phabricator.wikimedia.org/T195193#4245187 (10Marostegui) [08:31:37] 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy - https://phabricator.wikimedia.org/T190704#4245194 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` ['db2... [08:32:52] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Epic: Make wb_terms table fancy - https://phabricator.wikimedia.org/T188992#4245207 (10Ladsgroup) [08:32:56] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-Ladsgroup, 10Wikidata-Ministry-Of-Magic: Apply schema changes to an isolated database and examine the results - https://phabricator.wikimedia.org/T191391#4245205 (10Ladsgroup) 05Open>03Resolved This is done. Don't repool it unless i... [08:46:00] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-Ladsgroup, 10Wikidata-Ministry-Of-Magic: Apply schema changes to an isolated database and examine the results - https://phabricator.wikimedia.org/T191391#4245229 (10jcrespo) > Don't repool it unless it's back to the old status. So, to... [08:46:09] Amir1: ^ [08:49:18] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.32-release-notes (WMF-deploy-2018-05-22 (1.32.0-wmf.5)), and 2 others: Clean up indexes of wb_terms table - https://phabricator.wikimedia.org/T194273#4245235 (10jcrespo) I thought that was the query that broke things, am I wrong? [08:51:30] jynus, marostegui: when you have a minute I'd like to find an agreement on Debmonitor MySQL connection params, if possible: https://gerrit.wikimedia.org/r/#/c/436286/ (moved from https://gerrit.wikimedia.org/r/#/c/436244/ ) [08:51:56] 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy - https://phabricator.wikimedia.org/T190704#4245249 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db2092.codfw.wmnet'] ``` and were **ALL** successful. [08:52:18] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-Ladsgroup, 10Wikidata-Ministry-Of-Magic: Apply schema changes to an isolated database and examine the results - https://phabricator.wikimedia.org/T191391#4103598 (10Marostegui) Yes, it is marked as not repool on the config [08:52:43] jynus: Yes [08:52:56] yes to which? [08:53:01] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-Ladsgroup, 10Wikidata-Ministry-Of-Magic: Apply schema changes to an isolated database and examine the results - https://phabricator.wikimedia.org/T191391#4245257 (10Ladsgroup) >>! In T191391#4245229, @jcrespo wrote: > So, to clarify, we... [08:53:17] I suppose T191391#4245257 [08:53:17] T191391: Apply schema changes to an isolated database and examine the results - https://phabricator.wikimedia.org/T191391 [08:56:45] 10DBA: reimage db2083 back into wikidata (s8) - https://phabricator.wikimedia.org/T196047#4245259 (10jcrespo) [08:57:01] 10DBA: reimage db2083 back into wikidata (s8) - https://phabricator.wikimedia.org/T196047#4245272 (10jcrespo) p:05Triage>03Normal [09:08:35] 10DBA, 10Patch-For-Review: reimage db2083 back into wikidata (s8) - https://phabricator.wikimedia.org/T196047#4245259 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jynus on sarin.codfw.wmnet for hosts: ``` ['db2083.codfw.wmnet'] ``` The log can be found in `/var/log/wmf-auto-reimage/2018053109... [09:18:44] yeah [09:19:16] jynus: marostegui I'm deleting 15M rows from logging table of commons (last bits of code to delete auto patrol logs than happened between 2007-2011) [09:19:28] wait [09:19:48] It's not needed for wikidata because it started in 2012 [09:19:52] don't do that [09:20:01] until s4 lag is gone [09:20:01] stopped, why? [09:20:06] okay [09:20:35] do you know how to see that? [09:20:57] I use https://grafana.wikimedia.org/dashboard/db/mysql-aggregated?orgId=1 [09:21:11] but it's one second :/ [09:21:23] Check codfw [09:21:36] https://grafana.wikimedia.org/dashboard/db/mysql-aggregated?orgId=1&panelId=6&fullscreen&var-dc=codfw%20prometheus%2Fops&var-group=core&var-shard=s4&var-role=All [09:21:42] oh boy [09:22:04] okay. Good to know [09:22:06] Thanks [09:30:29] 10DBA, 10Patch-For-Review: reimage db2083 back into wikidata (s8) - https://phabricator.wikimedia.org/T196047#4245364 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db2083.codfw.wmnet'] ``` and were **ALL** successful. [10:06:30] 10DBA, 10ContentTranslation, 10Data-Services: Consider adding ContentTranslation (CX) tables to wiki replicas - https://phabricator.wikimedia.org/T196020#4245485 (10Nikerabbit) We will need to investigate the implications before doing this. [12:56:54] 10DBA, 10Math, 10Operations: Remove table `math` from the database - https://phabricator.wikimedia.org/T196055#4245815 (10Reedy) [13:12:56] 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy - https://phabricator.wikimedia.org/T190704#4245853 (10Marostegui) labsdb1009 has been moved under the new sanitarium hosts. We will leave it replicating till Monday befor... [13:13:16] 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy - https://phabricator.wikimedia.org/T190704#4245854 (10Marostegui) [13:17:07] 10DBA, 10Math, 10Operations: Remove table `math` from the database - https://phabricator.wikimedia.org/T196055#4245860 (10Marostegui) [13:17:12] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#4245859 (10Marostegui) [15:31:54] 10DBA, 10Patch-For-Review: reimage db2083 back into wikidata (s8) - https://phabricator.wikimedia.org/T196047#4246209 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jynus on sarin.codfw.wmnet for hosts: ``` ['db2079.codfw.wmnet'] ``` The log can be found in `/var/log/wmf-auto-reimage/2018053115... [15:54:37] 10DBA, 10Patch-For-Review: reimage db2083 back into wikidata (s8) - https://phabricator.wikimedia.org/T196047#4246262 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db2079.codfw.wmnet'] ``` and were **ALL** successful. [16:31:59] 10DBA, 10Patch-For-Review: Setup tendril database monitoring on 2 new hosts, one on eqiad and one on codfw - https://phabricator.wikimedia.org/T184704#4246361 (10RobH) [22:47:26] 10DBA, 10MediaWiki-User-management, 10Anti-Harassment (AHT Sprint 21/22): Draft a proposal for granular blocks table schema(s), submit for DBA review - https://phabricator.wikimedia.org/T193449#4247505 (10dmaza) Discussion here T104099 makes my argument stronger for Option A