[09:23:24] I cannot find the staging database, on which host or port it was? [09:23:47] is it 3350 ? [09:23:53] yep [09:24:12] on dbstore1005:3350 [09:26:00] thanks, I added it to the scripts in case it would need backups in the future [09:26:07] thank you! [10:40:43] the orchestration part on https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/494899/9/modules/profile/files/mariadb/daily_snapshot.py is almost finished [10:41:06] that will read a snapshot config file and run transfer.py + remote prepare [10:41:38] oh nice! [10:41:46] is it ready for a review? [10:41:58] not yet finished, but I would start looking at it [10:42:11] cool, will do then [10:42:14] so anything high level should be ready. although [10:42:17] I see the staging already <3 [10:42:31] I have not yet documented the commit either [10:42:35] this will run in cumin (or similar)? [10:42:43] may need some explanations to undertand the idea [10:42:48] that is literally the missing part [10:42:58] ah ok haha [10:43:04] execute_remotelly [10:43:25] not sure if to use cumin, or my own wrapper of cumin, or something else [10:43:59] yeah, I mean the host that will run it [10:44:19] also I would like to see transfer.py reimplemented by volans for a more generic usage and future extensibility [10:45:16] so please start any high level ideas or obvious errors like typos, but it is not yet finished [10:46:14] sure :) [10:46:21] (thinks like logging, error handling, etc not yet done [11:15:52] 10DBA, 10Analytics, 10Operations, 10ops-eqiad, and 2 others: rack/setup/install labsdb1012.eqiad.wmnet - https://phabricator.wikimedia.org/T215231 (10Marostegui) 05Open→03Resolved As per our earlier chat - this seems to be working fine after the puppet change to get the FW opened for labsdb1012 [11:38:23] marostegui - hola - could you tell me what's [XIJUAwpAMFgAALH6YU0AAABY] 2019-03-08 11:37:43: Excepción grave de tipo "Wikimedia\Rdbms\DBTransactionSizeError" about? [11:38:38] I was deleting a page in ky.wikibooks [11:38:41] a broken redirect [11:38:44] let me see [11:38:46] not many revisions, etc. [11:38:51] sure, thanks [11:40:28] Transaction spent 3.6647782325745 second(s) in writes, exceeding the limit of 3. [11:40:51] maybe it was a punctual error? can you try again? [11:41:04] this time it went fine [11:41:10] puntual error apparently [11:41:18] :) [11:41:25] sorry for disturbing then [11:41:45] I got few minutes before an Error 503 so I thought it was related [11:41:47] you were not disturbing :) [11:42:11] I'm glad I'm not :) [14:13:32] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping page.page_no_title_convert on wmf databases - https://phabricator.wikimedia.org/T86342 (10Marostegui) [14:13:50] 10DBA, 10Schema-change, 10Tracking: [DO NOT USE] Schema changes for Wikimedia wikis (tracking) [superseded by #Blocked-on-schema-change] - https://phabricator.wikimedia.org/T51188 (10Marostegui) [14:13:52] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping page.page_no_title_convert on wmf databases - https://phabricator.wikimedia.org/T86342 (10Marostegui) 05Open→03Resolved This is all done! One less drift between HEAD and production [14:15:42] 10DBA, 10Growth-Team, 10Notifications, 10Schema-change: Remove etp_user from echo_target_page in production - https://phabricator.wikimedia.org/T217453 (10Marostegui) This required some testing as x1 runs ROW based replication, so we need to be careful as dropping a column might break replication if not ex... [14:21:06] 10DBA, 10Growth-Team, 10Notifications, 10Schema-change: Remove etp_user from echo_target_page in production - https://phabricator.wikimedia.org/T217453 (10jcrespo) We can do the second at a period of low traffic, although we should do some testing to make sure it doesn't break things. [14:31:12] 10DBA, 10Growth-Team, 10Notifications, 10Schema-change: Remove etp_user from echo_target_page in production - https://phabricator.wikimedia.org/T217453 (10Marostegui) >>! In T217453#5011175, @jcrespo wrote: > We can do the second at a period of low traffic, although we should do some testing to make sure i... [14:32:56] 10DBA, 10Growth-Team, 10Notifications, 10Schema-change: Remove etp_user from echo_target_page in production - https://phabricator.wikimedia.org/T217453 (10jcrespo) +1 [14:34:45] 10DBA, 10Growth-Team, 10Notifications, 10Schema-change: Remove etp_user from echo_target_page in production - https://phabricator.wikimedia.org/T217453 (10Marostegui) 05Open→03Stalled [14:36:30] 10DBA, 10Growth-Team, 10Notifications, 10Schema-change: Remove etp_user from echo_target_page in production - https://phabricator.wikimedia.org/T217453 (10jcrespo) Maybe rather than stall it (or in addition), you can block it on a -presumably new- task to purchase such host? [14:37:36] 10DBA, 10Growth-Team, 10Notifications, 10Schema-change: Remove etp_user from echo_target_page in production - https://phabricator.wikimedia.org/T217453 (10Marostegui) Those replacements are already in place, we are just waiting for them to be racked: {T211613}