[05:26:12] 10DBA, 10MediaWiki-API: prop=revisions with rvdir=newer gives internal_api_error_DBQueryError for page with many version - https://phabricator.wikimedia.org/T108968 (10Marostegui) 05Open>03Resolved This is now using the `page_timestamp` for all the hosts in s5. Probably this has been fixed with all the mul... [07:03:58] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Marostegui) @Bstorm these changes have been replicated to s4 and s7, please re-run the views when you get a chance. Thank you! [07:17:42] 10DBA, 10Data-Services, 10cloud-services-team, 10wikitech.wikimedia.org: Move wikitech and labstestwiki to s5 - https://phabricator.wikimedia.org/T167973 (10jcrespo) Cross-dc repos- I would try to avoid them unless we are in an emergency- I have 2 options, either create a temporary new section for wikitech... [07:21:46] 10DBA, 10Data-Services, 10cloud-services-team, 10wikitech.wikimedia.org: Move wikitech and labstestwiki to s5 - https://phabricator.wikimedia.org/T167973 (10Marostegui) I would prefer option 2, import wikitech to s5. However, I am not sure how many blockers we have along the way and if we could get them re... [07:24:34] 10DBA, 10Data-Services, 10cloud-services-team, 10wikitech.wikimedia.org: Move wikitech and labstestwiki to s5 - https://phabricator.wikimedia.org/T167973 (10jcrespo) We can do it on switchback- that is why I suggested to create a temporary host for wikitech. Of course, that will increase the chances of a s... [07:25:40] 10DBA, 10Data-Services, 10cloud-services-team, 10wikitech.wikimedia.org: Move wikitech and labstestwiki to s5 - https://phabricator.wikimedia.org/T167973 (10Marostegui) Yeah, I am not sure if I would prefer a split brain or cross-dc queries only for wikitech (that's why I suggested to change db-codfw.php t... [09:26:53] please have a look at https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/457847/1/wmf-config/db-codfw.php [09:27:03] this is a first pass, but we need more attention to it [09:28:13] for some reason, some hosts provide bad query plans, I am researching on db1114 so we can apply it to codfw hosts if necessary [09:28:49] This is the host that gave issues some months ago [09:29:25] it uses the wrong index on queries, unlike most other hosts (except codfw master) [09:29:57] but focus for now on the codfw CR [09:31:02] ok [09:31:03] checking [09:31:12] I checked, and most api on codfw do the right thing [09:37:38] meanwhile, I am going to deploy the query killer to all core hosts [10:25:34] so db2088:3311 and db2085:3311 are missing its partitioning on logging [10:26:00] :( [10:26:03] good catch [10:26:15] Shall I get on to those? [10:26:18] so far [10:26:21] I am checking all [10:26:28] these were the first 2 I checked [10:26:40] ok, let me know how it ends up [10:26:43] And I can get those fixed [10:47:49] 10DBA, 10Operations, 10Epic, 10Patch-For-Review: DB meta task for next DC failover issues - https://phabricator.wikimedia.org/T189107 (10jcrespo) Missing partitions on codfw: ``` db2085:3311:enwiki:logging db2088:3311:enwiki:logging db2088:3312:bgwiktionary:revision db2088:3312:bgwiktionary:logging db2088... [10:49:17] marostegui: how long will you be away, I am not sure brad will arrive to the one scheduled [10:50:59] I will leave also some screens partitioning enwiki on codfw unless you tell me not to [11:04:58] go for it [11:05:13] jynus: My meeting is from 16:15 to 17:00 [11:07:06] I will delay it until 17:20 ? [11:07:16] is it too late? [11:07:17] or 17:15 if you want [11:07:19] ok [11:07:19] no, fine by me [11:07:28] My other meeting finishes at 17:00 [11:10:19] 10DBA, 10Operations, 10Epic, 10Patch-For-Review: DB meta task for next DC failover issues - https://phabricator.wikimedia.org/T189107 (10Marostegui) I have checked bgwitionary, eowiki, idwiki and frwiktionary and they do not exist on eqiad either. [11:35:38] 10DBA, 10Data-Services, 10cloud-services-team, 10wikitech.wikimedia.org: Move wikitech and labstestwiki to s5 - https://phabricator.wikimedia.org/T167973 (10Marostegui) There is stuff that constantly tries to write to db2037 (even now) but fails because of read-only: https://logstash.wikimedia.org/goto/bd4... [12:27:42] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Bstorm) Done! [12:29:24] 10DBA, 10Operations, 10Epic, 10Patch-For-Review: DB meta task for next DC failover issues - https://phabricator.wikimedia.org/T189107 (10Marostegui) I have checked that no codfw hosts have notifications disabled on puppet or on icinga itself. [12:39:36] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Marostegui) [12:40:27] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_moved_to_title/rc_moved_to_ns on wmf databases - https://phabricator.wikimedia.org/T51191 (10Marostegui) [12:40:34] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_cur_time on wmf databases - https://phabricator.wikimedia.org/T67448 (10Marostegui) [12:41:10] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_cur_time on wmf databases - https://phabricator.wikimedia.org/T67448 (10Marostegui) s3 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1002 [] db1124 [] db1123 [] db1095 [] db1078 [] db1077 [] db1075 [12:41:14] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_moved_to_title/rc_moved_to_ns on wmf databases - https://phabricator.wikimedia.org/T51191 (10Marostegui) s3 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1002 [] db1124 [] db1123 [] db1095 [] db1078 []... [12:41:17] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Marostegui) s3 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1002 [] db1124 [] db1123 [] db1095 [] db1078 [] db1077 [] db1075 [12:41:35] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Marostegui) [12:41:48] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_moved_to_title/rc_moved_to_ns on wmf databases - https://phabricator.wikimedia.org/T51191 (10Marostegui) [12:42:01] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_cur_time on wmf databases - https://phabricator.wikimedia.org/T67448 (10Marostegui) [12:44:25] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_cur_time on wmf databases - https://phabricator.wikimedia.org/T67448 (10Marostegui) [12:44:33] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_moved_to_title/rc_moved_to_ns on wmf databases - https://phabricator.wikimedia.org/T51191 (10Marostegui) [13:59:41] 10DBA, 10Data-Services: Prepare and check storage layer for fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T202820 (10Marostegui) 05stalled>03Open [14:07:52] 10DBA, 10JADE, 10Operations, 10TechCom-RFC, 10Scoring-platform-team (Current): Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10Halfak) I think that querying by within-judgement content should be very limited (and probably within the... [14:08:39] 10DBA, 10Patch-For-Review: Add support for socket path and/or port (multiinstance support) to redact_sanitarium.sh - https://phabricator.wikimedia.org/T203394 (10Banyek) 05Open>03Resolved root@db2094:~# /usr/local/sbin/redact_sanitarium.sh -d fixcopyrightwiki -S /run/mysqld/mysqld.s3.sock -- abuse_filter_l... [14:11:34] 10DBA, 10Data-Services: Prepare and check storage layer for fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T202820 (10Marostegui) In order to bypass the `access denied` typical error due to the mariadb bug, I have created the database (_p) and granted the `labsdb` user access to that db. ``` fo... [15:13:49] 10DBA, 10Wikimedia-Site-requests, 10Core-Platform-Team (CPT-Q1-Jul-Sep-2018), 10Patch-For-Review: advisorswiki is not in any s?.dblist - https://phabricator.wikimedia.org/T202904 (10Anomie) 05Open>03Resolved [15:14:18] 10DBA, 10Wikimedia-Site-requests, 10Core-Platform-Team (CPT-Q1-Jul-Sep-2018), 10Patch-For-Review: advisorswiki is not in any s?.dblist - https://phabricator.wikimedia.org/T202904 (10Anomie) Yes, the other two checkboxes are done by my patch. [16:25:12] 10DBA, 10Data-Services: Prepare and check storage layer for fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T202820 (10Bstorm) @Marostegui, on all three replicas, I just got: ``` pymysql.err.InternalError: (1290, 'The MariaDB server is running with the --read-only option so it cannot execute th... [16:26:45] 10DBA, 10Data-Services: Prepare and check storage layer for fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T202820 (10Marostegui) We haven't changed anything from our side as far as I know - anything changed from your side? [16:27:56] 10DBA, 10Data-Services: Prepare and check storage layer for fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T202820 (10Bstorm) Nope. It could have just been missed in the blur of GRANT changes? I thought we'd done one of these since, though. [16:29:44] 10DBA, 10Data-Services: Prepare and check storage layer for fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T202820 (10Bstorm) SHOW GRANTS doesn't show the SUPER option on that user, though. Huh. [16:30:51] 10DBA, 10Data-Services: Prepare and check storage layer for fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T202820 (10Marostegui) Yeah, not sure what happened, it is supposed to have it: https://github.com/wikimedia/puppet/blob/production/modules/role/templates/mariadb/grants/wiki-replicas.sql#... [16:33:40] 10DBA, 10Data-Services: Prepare and check storage layer for fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T202820 (10Bstorm) Works! views and indexes are in place. Setting up DNS and so forth. [17:13:21] 10DBA, 10Data-Services: Prepare and check storage layer for fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T202820 (10Bstorm) Noted for my reference: ``` requests.exceptions.ConnectionError: HTTPConnectionPool(host='cloudservices1003.wikimedia.org', port=9001): Max retries exceeded with url:... [17:39:53] 10DBA, 10Data-Services: Prepare and check storage layer for fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T202820 (10Bstorm) 05Open>03Resolved a:03Bstorm Updated docs to get around the above problem. All steps are now done for cloud services. I am able to connect from tools. [19:51:44] 10DBA, 10Wiki-Loves-Monuments-Database: mysqldump is timing out preventing all tables from being included in the dump - https://phabricator.wikimedia.org/T138517 (10JeanFred) This is still happening: ``` 2018-09-04_19:24:21 Dump database... mysqldump: Error 2013: Lost connection to MySQL server during query w... [20:05:32] 10DBA, 10Wiki-Loves-Monuments-Database: mysqldump is timing out preventing all tables from being included in the dump - https://phabricator.wikimedia.org/T138517 (10Marostegui) Can you try the mysqldump adding `--skip-extended-insert` to the original command? This will make the data dump and the load slower,...