[05:15:03] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1073 - https://phabricator.wikimedia.org/T194197#4193082 (10Marostegui) 05Open>03Resolved a:03Cmjohnson The disk is now part of the RAID again. ``` root@db1073:~# megacli -PDRbld -ShowProg -PhysDrv [32:7] -aALL Device(Encl-32 Slot-7) is not in rebui... [09:08:52] 10DBA, 10Patch-For-Review: Productionize 8 eqiad hosts - https://phabricator.wikimedia.org/T192979#4193349 (10jcrespo) This is a proposal for the remaining decommission/setup: ``` m1: master: db1063 (C5) replica: db1117:3331 (A8) m2: master: db1065 (D1) replica: db1117:3332 (A8) m3: master: db1072 (B2) repli... [09:45:39] 10DBA, 10Commons, 10MediaWiki-Watchlist, 10Wikidata, and 4 others: Re-enable Wikidata Recent Changes integration on Commons - https://phabricator.wikimedia.org/T179010#4193388 (10Aklapper) 05stalled>03Open [09:45:50] 10DBA, 10Commons, 10Contributors-Team, 10MediaWiki-Watchlist, and 12 others: "Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis - https://phabricator.wikimedia.org/T171027#4193389 (10Aklapper) [09:52:09] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-Ladsgroup, 10Wikidata-Ministry-Of-Magic: Apply schema changes to an isolated database and examine the results - https://phabricator.wikimedia.org/T191391#4193400 (10Ladsgroup) After running the optimize table we have this now: ``` test_... [09:52:54] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-Ladsgroup, 10Wikidata-Ministry-Of-Magic: Apply schema changes to an isolated database and examine the results - https://phabricator.wikimedia.org/T191391#4193415 (10jcrespo) Yes, that is much better. [09:57:58] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-Ladsgroup, 10Wikidata-Ministry-Of-Magic: Apply schema changes to an isolated database and examine the results - https://phabricator.wikimedia.org/T191391#4103598 (10Ladsgroup) I think this part is done, I make some actionables. [10:37:50] 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy - https://phabricator.wikimedia.org/T190704#4193603 (10Marostegui) To sum up the last events. db1116:3313 yesterday broke with: ``` Last_IO_Error: Got fatal error 1236 fr... [10:39:06] 10DBA, 10Patch-For-Review: Productionize 8 eqiad hosts - https://phabricator.wikimedia.org/T192979#4193606 (10Marostegui) >>! In T192979#4193349, @jcrespo wrote: > This is a proposal for the remaining decommission/setup: > > > ``` > m1: master: db1063 (C5) replica: db1117:3331 (A8) > m2: master: db1065 (D1)... [10:40:27] 10DBA, 10Patch-For-Review: Productionize 8 eqiad hosts - https://phabricator.wikimedia.org/T192979#4193620 (10jcrespo) What is the state of db1116 ? I would like to move its master so I can clone db1072 away. [10:41:47] 10DBA, 10Patch-For-Review: Productionize 8 eqiad hosts - https://phabricator.wikimedia.org/T192979#4193626 (10Marostegui) >>! In T192979#4193620, @jcrespo wrote: > What is the state of db1116 ? I would like to move its master so I can clone db1072 away. db1116 is not in use. It is just replicating from s3 (db... [10:54:39] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2067 - https://phabricator.wikimedia.org/T194103#4193662 (10Marostegui) a:05Marostegui>03Papaul [11:08:14] 10DBA, 10Wiki-Loves-Monuments-Database: mysqldump is timing out preventing all tables from being included in the dump - https://phabricator.wikimedia.org/T138517#2402642 (10Marostegui) @Lokal_Profil is there anything pending here? [11:12:59] 10DBA, 10Patch-For-Review: Productionize 8 eqiad hosts - https://phabricator.wikimedia.org/T192979#4193688 (10jcrespo) Blocked on depooling db1072 as snapshots are running on it. [11:17:30] 10DBA, 10Patch-For-Review: Productionize 8 eqiad hosts - https://phabricator.wikimedia.org/T192979#4193706 (10jcrespo) [11:17:55] 10DBA, 10Patch-For-Review: Productionize 8 eqiad hosts - https://phabricator.wikimedia.org/T192979#4156169 (10jcrespo) [12:04:20] 10DBA, 10Patch-For-Review: install and puppetize memcached on dbmonitor hosts (tendril frontends) - https://phabricator.wikimedia.org/T133906#4193767 (10Marostegui) >>! In T133906#4000472, @gerritbot wrote: > Change 414502 **merged** by Jcrespo: > [operations/puppet@production] tendril: Add memcache to tendril... [13:02:40] 10DBA, 10Patch-For-Review: install and puppetize memcached on dbmonitor hosts (tendril frontends) - https://phabricator.wikimedia.org/T133906#4193907 (10jcrespo) ``` dbmonitor1001:~$ echo stats | nc 127.0.0.1 11211 STAT pid 459 STAT uptime 1228445 STAT time 1525870917 STAT version 1.4.21 STAT libevent 2.0.21-s... [13:03:55] 10DBA, 10Patch-For-Review: install and puppetize memcached on dbmonitor hosts (tendril frontends) - https://phabricator.wikimedia.org/T133906#4193909 (10jcrespo) 05Open>03Resolved a:03jcrespo db1011 doesn't exist anymore. [13:56:56] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: Drop 'tmp1' index from wb_terms table in production - https://phabricator.wikimedia.org/T194270#4194080 (10Ladsgroup) [14:01:31] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: Clean up indexes of wb_terms table - https://phabricator.wikimedia.org/T194273#4194132 (10Ladsgroup) [14:01:48] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Epic: Make wb_terms table fancy - https://phabricator.wikimedia.org/T188992#4194153 (10Ladsgroup) [14:15:11] 10DBA, 10Operations, 10Traffic: Framework to transfer files over the LAN - https://phabricator.wikimedia.org/T156462#4194193 (10Rduran) hpenc looks interesting, so maybe we can keep it in mind for future improvements. [14:16:44] 10DBA, 10Operations, 10Traffic: Framework to transfer files over the LAN - https://phabricator.wikimedia.org/T156462#4194198 (10jcrespo) Yes, I was not suggesting to do it now, just document the suggestion for the future- or maybe they can even set it up for us in parallel. Changing the algorithm, assuming o... [14:24:48] 10DBA, 10Operations, 10Traffic: Framework to transfer files over the LAN - https://phabricator.wikimedia.org/T156462#4194211 (10Vgutierrez) in stretch chacha20 is available as "chacha20" and in jessie as "chacha20-poly1305", BTW for big enough block size (16384 bytes), chacha20 performs better than rc4 on on... [14:35:56] 10DBA, 10Operations, 10Traffic: Framework to transfer files over the LAN - https://phabricator.wikimedia.org/T156462#4194233 (10Rduran) Thank you both! I'm using "chacha20" right now and it seems to work just fine (I'm using buster, but stretch is also on 1.1.0). Does jessie need to be supported too? [14:36:31] 10DBA, 10Operations, 10Traffic: Framework to transfer files over the LAN - https://phabricator.wikimedia.org/T156462#4194234 (10jcrespo) No, stick to stretch, that is ok- that is the target. [15:03:21] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: Clean up indexes of wb_terms table - https://phabricator.wikimedia.org/T194273#4194297 (10Marostegui) p:05Triage>03Normal Thanks! We can get rid of them once the patch is submitted and merged. [15:14:42] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: Drop 'tmp1' index from wb_terms table in production - https://phabricator.wikimedia.org/T194270#4194338 (10Marostegui) p:05Triage>03Normal As I have to do a schema change on s8 soonish I wanted to include this one too. But I wanted to make s... [15:21:57] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: Drop 'tmp1' index from wb_terms table in production - https://phabricator.wikimedia.org/T194270#4194355 (10Lucas_Werkmeister_WMDE) As far as I can tell, the index is not referenced anywhere ([code search](https://codesearch.wmflabs.org/search/?q=t... [15:23:26] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: Drop 'tmp1' index from wb_terms table in production - https://phabricator.wikimedia.org/T194270#4194357 (10jcrespo) This would have been very useful in this case https://dev.mysql.com/doc/refman/8.0/en/invisible-indexes.html [18:47:29] 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Wikidata-Ministry-Of-Magic: Schema change for new change_tag_def - https://phabricator.wikimedia.org/T194302#4194896 (10Ladsgroup) [18:56:34] 10DBA, 10Data-Services, 10Dumps-Generation, 10MediaWiki-Platform-Team, 10Patch-For-Review: Configure Toolforge replica views and dumps for the new MCR tables - https://phabricator.wikimedia.org/T184446#4194935 (10Bstorm) 05Open>03Resolved a:03Bstorm [19:03:10] 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Wikidata-Ministry-Of-Magic: Schema change for new change_tag_def - https://phabricator.wikimedia.org/T194302#4194971 (10jcrespo) CREATE TABLEs are not dangerous and they should not be blocked by #DBAs (any deployer can handle t... [20:03:42] 10DBA, 10AbuseFilter, 10Wikimedia-Site-requests: Run listed queries for all wikis in order to unblock existing changes - https://phabricator.wikimedia.org/T193894#4195147 (10Daimona) @Marostegui, sorry again, could you please run the following? ``` SELECT af_id FROM abuse_filter WHERE af_pattern RLIKE '\\\\...