[00:35:20] <wikibugs>	 10DBA, 10Operations, 10Availability (Multiple-active-datacenters), 10Patch-For-Review, 10Performance-Team (Radar): Make apache/maintenance hosts TLS connections to mariadb work - https://phabricator.wikimedia.org/T175672#3599592 (10tstarling) >>! In T175672#3635865, @aaron wrote: >>>! In T175672#3635140,...
[01:33:16] <wikibugs>	 10DBA, 10Operations, 10Availability (Multiple-active-datacenters), 10Patch-For-Review, 10Performance-Team (Radar): Make apache/maintenance hosts TLS connections to mariadb work - https://phabricator.wikimedia.org/T175672#3749801 (10tstarling) >>! In T175672#3655525, @aaron wrote: > I forget to mention, u...
[06:46:43] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017): Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3750025 (10Marostegui)
[06:51:07] <wikibugs>	 10DBA, 10Analytics-Kanban, 10Operations, 10ops-eqiad, 10User-Elukey: db1046 BBU looks faulty - https://phabricator.wikimedia.org/T166141#3750027 (10Marostegui) After the BBU re-learn: ``` ˜/icinga-wm 7:50> RECOVERY - MegaRAID on db1046 is OK: OK: optimal, 1 logical, 2 physical, WriteBack policy  ```
[07:01:21] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, and 2 others: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3750032 (10Marostegui)
[07:01:35] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, and 2 others: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3490050 (10Marostegui) 05Open>03Resolved s5 primary master done
[07:02:26] <wikibugs>	 10DBA, 10Data-Services, 10Quarry, 10Patch-For-Review: Raise concurrent mysql connection limit for Quarry (or throttle application concurrency) - https://phabricator.wikimedia.org/T180141#3750037 (10Marostegui) 05Open>03Resolved a:03Marostegui Done.
[07:06:02] <wikibugs>	 10DBA, 10Schema-change: Drop ca_repo_author index from mediawikiwiki.code_authors - https://phabricator.wikimedia.org/T180165#3750040 (10Marostegui) 05Open>03Resolved a:03Marostegui This table is tiny, so I just did it.
[07:36:07] <wikibugs>	 10DBA, 10Cloud-Services, 10User-Jayprakash12345, 10cloud-services-team (Kanban): Prepare and check storage layer for hif.wiktionary - https://phabricator.wikimedia.org/T173647#3750051 (10Marostegui) Anything pending on this task?
[07:38:19] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: db2059 storage crash - https://phabricator.wikimedia.org/T180196#3750056 (10Marostegui)
[07:38:29] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: db2059 storage crash - https://phabricator.wikimedia.org/T180196#3750068 (10Marostegui) 05Open>03Resolved a:03Marostegui
[07:46:36] <wikibugs>	 10DBA, 10Schema-change: Drop ep_articles_course_id index - https://phabricator.wikimedia.org/T180166#3750076 (10Marostegui)
[07:55:07] <wikibugs>	 10DBA, 10Patch-For-Review, 10Schema-change: Drop thread.thread_root_2 indexes - https://phabricator.wikimedia.org/T180164#3750078 (10Marostegui)
[07:58:39] <wikibugs>	 10DBA, 10Schema-change: Drop ep_articles_course_id index - https://phabricator.wikimedia.org/T180166#3750081 (10Marostegui)
[07:59:29] <wikibugs>	 10DBA, 10Schema-change: Drop ep_articles_course_id index - https://phabricator.wikimedia.org/T180166#3749095 (10Marostegui)
[08:01:26] <wikibugs>	 10DBA, 10Schema-change: Drop ep_articles_course_id index - https://phabricator.wikimedia.org/T180166#3750083 (10Marostegui)
[08:02:10] <wikibugs>	 10DBA, 10Schema-change: Drop ep_articles_course_id index - https://phabricator.wikimedia.org/T180166#3749095 (10Marostegui)
[08:03:20] <wikibugs>	 10DBA, 10Schema-change: Drop ep_articles_course_id index - https://phabricator.wikimedia.org/T180166#3750085 (10Marostegui)
[08:03:38] <wikibugs>	 10DBA, 10Schema-change: Drop ep_articles_course_id index - https://phabricator.wikimedia.org/T180166#3749095 (10Marostegui) 05Open>03Resolved a:03Marostegui The tables were super small, so I went ahead and drop the index.
[08:14:58] <wikibugs>	 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3750091 (10Marostegui) db1105.s2 is now replicating
[08:33:58] <wikibugs>	 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3750100 (10Marostegui) db1105.s1 is now replicating
[08:34:10] <wikibugs>	 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3750101 (10Marostegui)
[09:06:26] <wikibugs>	 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: Drop the "wb_terms.wb_terms_language" index - https://phabricator.wikimedia.org/T179106#3750118 (10Marostegui) a:03Marostegui
[09:08:24] <wikibugs>	 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: Drop the "wb_terms.wb_terms_language" index - https://phabricator.wikimedia.org/T179106#3713408 (10Marostegui) I will definitely try to optimize the table, as it is getting out of hand: `-rw-rw---- 1 mysql mysql 716G Nov 10 09:08 /srv/sqldata/wiki...
[09:26:44] <wikibugs>	 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: Consider dropping the "wb_items_per_site.wb_ips_site_page" index - https://phabricator.wikimedia.org/T179793#3750128 (10Marostegui) a:03Marostegui I will start working on T179106 so I will also get this done while depooling servers.
[09:28:29] <wikibugs>	 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: Consider dropping the "wb_items_per_site.wb_ips_site_page" index - https://phabricator.wikimedia.org/T179793#3750131 (10Marostegui) p:05Triage>03Normal
[09:32:05] <wikibugs>	 10DBA, 10Cloud-Services, 10User-Jayprakash12345, 10cloud-services-team (Kanban): Prepare and check storage layer for hif.wiktionary - https://phabricator.wikimedia.org/T173647#3750135 (10aborrero) the DNS thing:  ```lines=20 aborrero@labcontrol1001:~$ source <(sudo cat ~root/novaenv.sh) aborrero@labcontrol...
[09:36:56] <wikibugs>	 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: Drop the "wb_terms.wb_terms_language" index - https://phabricator.wikimedia.org/T179106#3750159 (10Marostegui)
[09:56:16] <wikibugs>	 10DBA, 10Cloud-Services, 10User-Jayprakash12345, 10cloud-services-team (Kanban): Prepare and check storage layer for hif.wiktionary - https://phabricator.wikimedia.org/T173647#3750201 (10aborrero) Trying what @bd808 said about conntecting to the database:  ```lines=20 aborrero@tools-bastion-03:~$ sql hifwi...
[10:02:48] <wikibugs>	 10DBA, 10Cloud-Services, 10User-Jayprakash12345, 10cloud-services-team (Kanban): Prepare and check storage layer for hif.wiktionary - https://phabricator.wikimedia.org/T173647#3750229 (10Reedy) Needs my patch merging above
[10:09:35] <wikibugs>	 10DBA, 10Cloud-Services, 10User-Jayprakash12345, 10cloud-services-team (Kanban): Prepare and check storage layer for hif.wiktionary - https://phabricator.wikimedia.org/T173647#3750244 (10aborrero) Merged the patch:  ```lines=15 aborrero@puppetmaster1001:~$ sudo puppet-merge Fetching new commits from https:...
[10:25:20] <wikibugs>	 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: Drop the "wb_terms.wb_terms_language" index - https://phabricator.wikimedia.org/T179106#3750305 (10Marostegui) Going to test+optimize on db2089 which has wb_terms compressed. ``` -rw-rw---- 1 mysql mysql 529G Nov 10 10:25 /srv/sqldata.s5/wikidataw...
[10:28:57] <wikibugs>	 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: Drop the "wb_terms.wb_terms_language" index - https://phabricator.wikimedia.org/T179106#3750329 (10daniel) >>! In T179106#3750118, @Marostegui wrote: > I will definitely try to optimize the table, as it is getting out of hand  Now that we use Elas...
[10:57:11] <wikibugs>	 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: Drop the "wb_terms.wb_terms_language" index - https://phabricator.wikimedia.org/T179106#3750469 (10Marostegui) >>! In T179106#3750329, @daniel wrote: >>>! In T179106#3750118, @Marostegui wrote: >> I will definitely try to optimize the table, as it...
[10:59:24] <wikibugs>	 10DBA, 10Cloud-Services, 10User-Jayprakash12345, 10cloud-services-team (Kanban): Prepare and check storage layer for hif.wiktionary - https://phabricator.wikimedia.org/T173647#3750472 (10aborrero) Pending to follow-up with @madhuvishy
[11:22:11] <wikibugs>	 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: Drop the "wb_terms.wb_terms_language" index - https://phabricator.wikimedia.org/T179106#3750546 (10daniel) Interesting. I would have expected it to be a lot more than that.
[11:57:26] <wikibugs>	 10DBA, 10Schema-change: Drop thread.thread_root_2 indexes - https://phabricator.wikimedia.org/T180164#3750590 (10Reedy)
[12:35:13] <wikibugs>	 10DBA, 10Schema-change: Drop thread.thread_root_2 indexes - https://phabricator.wikimedia.org/T180164#3750664 (10Marostegui)
[12:38:05] <wikibugs>	 10DBA, 10Schema-change: Drop thread.thread_root_2 indexes - https://phabricator.wikimedia.org/T180164#3750665 (10Marostegui)
[12:40:51] <wikibugs>	 10DBA, 10Schema-change: Drop thread.thread_root_2 indexes - https://phabricator.wikimedia.org/T180164#3750672 (10Marostegui)
[12:41:21] <wikibugs>	 10DBA, 10Schema-change: Drop thread.thread_root_2 indexes - https://phabricator.wikimedia.org/T180164#3749062 (10Marostegui)
[12:41:30] <wikibugs>	 10DBA, 10Schema-change: Drop thread.thread_root_2 indexes - https://phabricator.wikimedia.org/T180164#3749062 (10Marostegui) 05Open>03Resolved a:03Marostegui
[12:41:40] <wikibugs>	 10DBA, 10Schema-change: Drop thread.thread_root_2 indexes - https://phabricator.wikimedia.org/T180164#3749062 (10Marostegui) Small tables - all dropped.
[13:16:55] <wikibugs>	 10DBA: Remove duplicate comment_temp indexes - https://phabricator.wikimedia.org/T180162#3750827 (10Anomie) While MediaWiki shouldn't ever insert rows that violate the "one comment_id per revision" constraint, it's helpful to check it at the database level too.
[13:48:49] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017): Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3750918 (10Marostegui)
[13:49:28] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017): Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3566137 (10Marostegui) All codfw is now done. I will start with eqiad next week, it will take...
[17:07:28] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017): Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3751610 (10Marostegui) >>! In T174569#3751540, @Stashbot wrote: > {nav icon=file, name=Mention...
[21:00:11] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017): Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3751907 (10Marostegui) The following hosts on s6 eqiad are done: dbstore1001 dbstore1002
[21:12:17] <wikibugs>	 10DBA, 10Cloud-Services, 10User-Jayprakash12345, 10cloud-services-team (Kanban): Prepare and check storage layer for hif.wiktionary - https://phabricator.wikimedia.org/T173647#3751917 (10madhuvishy) @aborrero and I caught up on this, and it looks like all the DNS records are created now:  ``` tools.ifttt-d...