[02:01:12] 10DBA, 10Jade, 10Operations, 10TechCom-RFC, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10Krinkle) For this weeks' TechCom-RFC Inbox triage, I'm unsure whether to move to Under discussion or Backlog. We usually move t... [07:03:10] 10DBA, 10MediaWiki-Configuration, 10Operations, 10Patch-For-Review, and 2 others: Create tool to handle the state of database configuration in MediaWiki in etcd - https://phabricator.wikimedia.org/T197126 (10ArielGlenn) [08:23:43] 10DBA: es2019 is not responsive - https://phabricator.wikimedia.org/T212833 (10Banyek) [08:23:55] 10DBA: es2019 is not responsive - https://phabricator.wikimedia.org/T212833 (10Banyek) p:05Triage→03Unbreak! [08:44:21] 10DBA, 10Patch-For-Review: es2019 is not responsive - https://phabricator.wikimedia.org/T212833 (10Banyek) [08:44:58] 10DBA, 10Patch-For-Review: es2019 is not responsive - https://phabricator.wikimedia.org/T212833 (10Banyek) p:05Unbreak!→03High I triage this as 'high' not unbreak, because the host wasn't in service [09:07:16] 10DBA, 10Patch-For-Review: es2019 is not responsive - https://phabricator.wikimedia.org/T212833 (10Banyek) according to https://wikitech.wikimedia.org/wiki/Platform-specific_documentation/Dell_PowerEdge_RN30 I reset the host with `racadm serveraction hardreset`, now the console is available [09:09:41] 10DBA, 10Analytics, 10Analytics-Kanban, 10User-Elukey: Review dbstore1002's non-wiki databases and decide which ones needs to be migrated to the new multi instance setup - https://phabricator.wikimedia.org/T212487 (10elukey) >>! In T212487#4839932, @elukey wrote: > > `project_illustration` was requested i... [09:10:13] 10DBA, 10Analytics, 10Analytics-Kanban, 10User-Elukey: Review dbstore1002's non-wiki databases and decide which ones needs to be migrated to the new multi instance setup - https://phabricator.wikimedia.org/T212487 (10elukey) Ok so as far as I can see this task seems waiting a few confirmations but if the p... [09:14:12] 10DBA, 10Patch-For-Review: es2019 is not responsive - https://phabricator.wikimedia.org/T212833 (10Banyek) after hard reset, I didn't find anything in the logs `/var/log/syslog` ` Jan 3 07:35:01 es2019 CRON[16225]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1) Jan 3 07:35:01 es2019 CRON[16... [09:14:32] 10DBA, 10Patch-For-Review: es2019 is not responsive - https://phabricator.wikimedia.org/T212833 (10Banyek) The mariadb started without any problem, and replication is resumed [09:14:58] 10DBA, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10elukey) As far as I can see we'd need to work on https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/479224/ right? [09:27:31] 10DBA, 10Patch-For-Review: es2019 is not responsive - https://phabricator.wikimedia.org/T212833 (10Banyek) [12:54:57] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [13:39:04] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [14:36:01] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [16:42:30] 10DBA, 10Jade, 10Operations, 10TechCom-RFC, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10Halfak) I'll be attending the meeting that @Krinkle mentions. But in the meantime, I'd like to offer an explanation for what Ja... [18:46:13] marostegui: o/ Can you help us unblock https://phabricator.wikimedia.org/T212154#4849981 as our service deployment is blocked on it? [19:05:32] 10DBA, 10SDC Engineering, 10SDC General, 10Wikidata, and 3 others: Create a production test wiki in group0 to parallel Wikimedia Commons - https://phabricator.wikimedia.org/T197616 (10Reedy) CC'ing #DBA for them to confirm whether we should use s3 or s4 [19:11:18] marostegui: ^ if you're about [19:27:20] 10DBA, 10Patch-For-Review: es2019 is not responsive - https://phabricator.wikimedia.org/T212833 (10Marostegui) 05Resolved→03Open Please run a full check of the tables to make sure data is ok Should be easy as there is only one table per DB [19:45:39] Reedy: he's out for the next few days, is that anything urgent? [21:51:43] 10DBA, 10Patch-For-Review: es2019 is not responsive - https://phabricator.wikimedia.org/T212833 (10Banyek) I'll start with this in the morning [22:44:51] 10DBA: dbstore1002 lagged on creating database - https://phabricator.wikimedia.org/T212908 (10Banyek) [22:45:00] 10DBA: dbstore1002 lagged on creating database - https://phabricator.wikimedia.org/T212908 (10Banyek) p:05Triage→03Normal