[05:42:44] 10DBA, 10AbuseFilter, 10Wikimedia-Site-requests: Run listed queries for all wikis in order to unblock existing changes - https://phabricator.wikimedia.org/T193894#4203544 (10Marostegui) >>! In T193894#4195147, @Daimona wrote: > @Marostegui, sorry again, could you please run the following? > > ``` > SELECT a... [05:43:01] 10DBA, 10AbuseFilter, 10Wikimedia-Site-requests: Run listed queries for all wikis in order to unblock existing changes - https://phabricator.wikimedia.org/T193894#4203545 (10Marostegui) 05Open>03Resolved a:03Marostegui [05:52:05] 10DBA, 10Patch-For-Review: Productionize 8 eqiad hosts - https://phabricator.wikimedia.org/T192979#4203548 (10Marostegui) >>! In T192979#4193349, @jcrespo wrote: > This is a proposal for the remaining decommission/setup: > > > ``` > m1: master: db1063 (C5) replica: db1117:3331 (A8) > m2: master: db1065 (D1)... [06:20:41] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#4203576 (10Marostegui) [06:20:43] 10DBA: Drop flaggedrevs tables on wikis where it is not enabled - https://phabricator.wikimedia.org/T174801#4203573 (10Marostegui) 05Open>03Resolved a:03Marostegui I have dropped the tables from testwiki too. Considering this task resolved. If in the future there are more wikis that get flaggedrev disabled... [06:20:45] 10DBA: Drop flaggedrevs tables on wikis where it is not enabled - https://phabricator.wikimedia.org/T174801#4203577 (10Marostegui) [06:24:15] 10DBA, 10Operations, 10decommission, 10ops-eqiad: Decommission db1029 and db1031 - https://phabricator.wikimedia.org/T184054#4203578 (10Marostegui) a:05RobH>03Cmjohnson Assigning it to Chris to reflect current status [06:37:29] 10DBA, 10AbuseFilter, 10Wikimedia-Site-requests: Run listed queries for all wikis in order to unblock existing changes - https://phabricator.wikimedia.org/T193894#4203586 (10Daimona) @Marostegui Ehm, unfortunately I can't see it. May it have a more relaxed policy? After all, being global abusefilter helper,... [06:40:45] 10DBA, 10AbuseFilter, 10Wikimedia-Site-requests: Run listed queries for all wikis in order to unblock existing changes - https://phabricator.wikimedia.org/T193894#4203594 (10Marostegui) >>! In T193894#4203586, @Daimona wrote: > @Marostegui Ehm, unfortunately I can't see it. May it have a more relaxed policy?... [06:57:52] 10DBA, 10AbuseFilter, 10Wikimedia-Site-requests: Run listed queries for all wikis in order to unblock existing changes - https://phabricator.wikimedia.org/T193894#4203621 (10Marostegui) @Daimona I have talked to @MoritzMuehlenhoff and @Legoktm to clarify a few things - can you access the paste now? I have ma... [07:10:41] 10DBA, 10AbuseFilter, 10Wikimedia-Site-requests: Run listed queries for all wikis in order to unblock existing changes - https://phabricator.wikimedia.org/T193894#4203642 (10Daimona) Yes, I confirm I can see it. Many thanks [07:56:06] going to deploy a schema change on s7 codfw with replication [07:57:04] ok [07:59:19] https://phabricator.wikimedia.org/T192979#4203548 [08:05:45] 10DBA, 10Patch-For-Review: Productionize 8 eqiad hosts - https://phabricator.wikimedia.org/T192979#4203765 (10jcrespo) You can use it with no problem, at least for the immediate future. [08:11:13] 10DBA, 10Patch-For-Review: Productionize 8 eqiad hosts - https://phabricator.wikimedia.org/T192979#4203771 (10Marostegui) Yeah, it will be just temporary till the new hosts arrive. Once they arrive we can stop db1116 and db1120 and copy the content to them. Do the failover from db1095 and db1102 to those two... [08:11:29] 10DBA, 10Patch-For-Review: Productionize 8 eqiad hosts - https://phabricator.wikimedia.org/T192979#4203772 (10Marostegui) [08:21:37] 10DBA, 10Patch-For-Review: Productionize 8 eqiad hosts - https://phabricator.wikimedia.org/T192979#4156670 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` db1120.eqiad.wmnet ``` The log can be found in `/var/log/wmf-auto-reimage/201805140821_mar... [08:27:21] 10DBA: Grant access to anomie to db1112 and db1112 (MCR tests hosts) - https://phabricator.wikimedia.org/T194626#4203806 (10Marostegui) [08:27:28] 10DBA: Grant access to anomie to db1112 and db1112 (MCR tests hosts) - https://phabricator.wikimedia.org/T194626#4203817 (10Marostegui) 05Open>03Resolved [08:39:40] 10DBA, 10Patch-For-Review: Productionize 8 eqiad hosts - https://phabricator.wikimedia.org/T192979#4203836 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db1120.eqiad.wmnet'] ``` and were **ALL** successful. [09:48:12] 10DBA: Decommission db1053 - https://phabricator.wikimedia.org/T194634#4204024 (10jcrespo) p:05Triage>03Normal [13:15:04] 10Blocked-on-schema-change, 10DBA, 10Multi-Content-Revisions, 10Patch-For-Review, 10User-Addshore: Change DEFAULT 0 for rev_text_id on production DBs - https://phabricator.wikimedia.org/T190148#4204520 (10Marostegui) [13:15:21] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10User-Ladsgroup, 10Wikidata-Ministry-Of-Magic: Schema change for rc_namespace_title_timestamp index - https://phabricator.wikimedia.org/T191519#4204521 (10Marostegui) [13:15:24] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10Patch-For-Review: Schema change for refactored actor storage - https://phabricator.wikimedia.org/T188299#4204522 (10Marostegui) [13:21:06] 10Blocked-on-schema-change, 10DBA, 10Multi-Content-Revisions, 10Patch-For-Review, 10User-Addshore: Change DEFAULT 0 for rev_text_id on production DBs - https://phabricator.wikimedia.org/T190148#4204533 (10Marostegui) s7 progress [] labsdb1009 [] labsdb1010 [] labsdb1011 [] db1102 [] dbstore1002 [] db107... [13:21:11] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10User-Ladsgroup, 10Wikidata-Ministry-Of-Magic: Schema change for rc_namespace_title_timestamp index - https://phabricator.wikimedia.org/T191519#4204534 (10Marostegui) s7 eqiad progress [] labsdb1009 [] labsdb1010 [] labsdb1011 [] db1102 [] dbstore1... [13:21:28] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10Patch-For-Review: Schema change for refactored actor storage - https://phabricator.wikimedia.org/T188299#4204535 (10Marostegui) s7 eqiad progress [] labsdb1009 [] labsdb1010 [] labsdb1011 [] db1102 []... [13:22:17] 10Blocked-on-schema-change, 10DBA, 10Multi-Content-Revisions, 10Patch-For-Review, 10User-Addshore: Change DEFAULT 0 for rev_text_id on production DBs - https://phabricator.wikimedia.org/T190148#4204537 (10Marostegui) [13:22:32] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10User-Ladsgroup, 10Wikidata-Ministry-Of-Magic: Schema change for rc_namespace_title_timestamp index - https://phabricator.wikimedia.org/T191519#4204538 (10Marostegui) [13:22:52] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10Patch-For-Review: Schema change for refactored actor storage - https://phabricator.wikimedia.org/T188299#4204539 (10Marostegui) [13:48:12] 10DBA, 10Operations, 10ops-eqiad: Move db1067 to row C - https://phabricator.wikimedia.org/T193835#4204590 (10Marostegui) This has been scheduled for Wed 16th [14:24:39] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2067 - https://phabricator.wikimedia.org/T194103#4204745 (10Papaul) a:05Papaul>03jcrespo @jcrespo disk replacement complete [14:25:46] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2067 - https://phabricator.wikimedia.org/T194103#4204748 (10Marostegui) Thanks! Let's see how it goes ``` physicaldrive 1I:1:10 (port 1I:box 1:bay 10, SAS, 600 GB, Rebuilding) ``` [14:51:56] 10DBA: Drop long_run_profiling unused table - https://phabricator.wikimedia.org/T194661#4204828 (10Marostegui) p:05Triage>03Normal [14:55:46] 10DBA: Drop long_run_profiling unused table - https://phabricator.wikimedia.org/T194661#4204855 (10Marostegui) Backup placed at: `dbstore1001:/srv/backups/tmp/T194661/enwiki_long_run_profiling.sql` [14:58:04] 10DBA: Drop long_run_profiling unused table - https://phabricator.wikimedia.org/T194661#4204868 (10Marostegui) 05Open>03Resolved Dropped [14:58:06] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#4204870 (10Marostegui) [14:59:04] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#3483121 (10Marostegui) [15:34:27] 10DBA: Drop unused tables: msg_resource msg_resource_links - https://phabricator.wikimedia.org/T194663#4204952 (10Marostegui) p:05Triage>03Normal [15:47:55] 10DBA: Drop unused tables: msg_resource msg_resource_links - https://phabricator.wikimedia.org/T194663#4204991 (10Marostegui) Backups left at: ``` dbstore1001:/srv/backups/tmp/T194663# ls s1 s2 s3 s4 s5 s6 s7 s8 ``` [19:20:57] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2067 - https://phabricator.wikimedia.org/T194103#4205552 (10Marostegui) 05Open>03Resolved This time it worked fine Thanks Papaul! ``` logicaldrive 1 (3.3 TB, RAID 1+0, OK) physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 600 GB, OK)...