[05:18:50] 10DBA, 10MediaWiki-extensions-UrlShortener, 10MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), 10Patch-For-Review, 10User-Ladsgroup: DBA review of UrlShortener - https://phabricator.wikimedia.org/T219777 (10Marostegui) ^ that was not meant to be for this table [07:02:53] 10Blocked-on-schema-change, 10Notifications, 10MW-1.33-notes (1.33.0-wmf.22; 2019-03-19), 10Patch-For-Review, 10Schema-change: Remove unused bundling DB fields - https://phabricator.wikimedia.org/T143763 (10Marostegui) 05Open→03Resolved This is all done! [07:04:59] 10Blocked-on-schema-change, 10DBA, 10Growth-Team, 10Notifications, 10Schema-change: Remove etp_user from echo_target_page in production - https://phabricator.wikimedia.org/T217453 (10Marostegui) a:03Marostegui This is similar (and on a smaller table) than T143763, so this can probably follow the same p... [07:05:27] 10Blocked-on-schema-change, 10DBA, 10Growth-Team, 10Notifications, 10Schema-change: Remove etp_user from echo_target_page in production - https://phabricator.wikimedia.org/T217453 (10Marostegui) 05Stalled→03Open [07:21:12] 10Blocked-on-schema-change, 10DBA, 10Growth-Team, 10Notifications, and 2 others: Remove etp_user from echo_target_page in production - https://phabricator.wikimedia.org/T217453 (10Marostegui) Altered the first 10 wikis, and will leave them for a few days just in case: ` aawiki aawikibooks aawiktionary abwi... [07:36:03] Morning jynus! [07:36:08] hi [07:36:26] dbstore2001 has had s8 repl stopped for the weekend, I saw it on Sat, but I didn't know if you were running tests on a loop or something, so didn't start it. Just FYI :) [08:52:16] I was doing testing, discover an issue (I think a puppet bug), I have restarted it again [08:52:28] great :) [14:22:56] 10DBA, 10MediaWiki-Configuration, 10Operations, 10Patch-For-Review, and 2 others: Create tool to handle the state of database configuration in MediaWiki in etcd - https://phabricator.wikimedia.org/T197126 (10CDanis) [15:42:26] 10DBA, 10Goal: Purchase and setup remaining hosts for database backups - https://phabricator.wikimedia.org/T213406 (10Cmjohnson) [15:42:44] \o/ [15:43:53] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install db1139|db1140.eqiad.wmnet (2 dump slaves) - https://phabricator.wikimedia.org/T218985 (10Marostegui) Those two hosts were added to site.pp and have been added to the install recipe a few days ago. [15:45:21] jynus: I have updated line 89 on the etherpad [15:45:26] to reflect that the eqiad ones arrived [16:32:14] I've added s3 maintenance to SOS [16:32:24] thanks [16:45:42] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install (5) codfw dedicated dump slaves - https://phabricator.wikimedia.org/T219463 (10jcrespo) [16:52:09] 10DBA, 10Operations, 10ops-eqiad: rack/setup/deploy eqiad dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T219399 (10jcrespo) Just to be clear, @Cmjohnson, of what we discussed on the meeting, for eqiad, this one (dbprov) is more important for us than those that arrived recen... [16:57:55] 10DBA, 10Reading-Infrastructure-Team-Backlog, 10WikimediaEditorTasks: Choose DB/Cluster for WikimediaEditorTasks tables - https://phabricator.wikimedia.org/T218302 (10Mholloway) [17:02:02] 10DBA, 10Goal: Purchase and setup remaining hosts for database backups - https://phabricator.wikimedia.org/T213406 (10jcrespo) [17:02:08] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install db1139|db1140.eqiad.wmnet (2 dump slaves) - https://phabricator.wikimedia.org/T218985 (10jcrespo) [18:17:53] 10DBA, 10Core Platform Team, 10MediaWiki-Redirects, 10MediaWiki-Revision-backend, 10Wikimedia-production-error: Unable to create redirect on dewiki - fatal DBQueryError - https://phabricator.wikimedia.org/T220353 (10Krinkle) [18:19:57] 10DBA, 10Core Platform Team, 10MediaWiki-Redirects, 10MediaWiki-Revision-backend, 10Wikimedia-production-error: Unable to create redirect on dewiki - fatal DBQueryError - https://phabricator.wikimedia.org/T220353 (10Krinkle) CC-ing DBA: Do we know of a change in schema, indexes or query traffic that coul... [19:45:21] 10DBA, 10Core Platform Team, 10MediaWiki-Redirects, 10MediaWiki-Revision-backend, 10Wikimedia-production-error: Unable to create redirect on dewiki - fatal DBQueryError - https://phabricator.wikimedia.org/T220353 (10Marostegui) The only schema change we have done today (and recently actually) has been to... [19:53:55] 10DBA, 10Core Platform Team, 10MediaWiki-Redirects, 10MediaWiki-Revision-backend, 10Wikimedia-production-error: Unable to create redirect on dewiki - fatal DBQueryError - https://phabricator.wikimedia.org/T220353 (10Marostegui) So this doesn't really like something "new" right? I mean, looks like it has... [20:01:09] 10DBA, 10Core Platform Team, 10MediaWiki-Redirects, 10MediaWiki-Revision-backend, 10Wikimedia-production-error: Unable to create redirect on dewiki - fatal DBQueryError - https://phabricator.wikimedia.org/T220353 (10Marostegui) Also no big changes or spikes on any graph on the s5 master (which serves dew...