[07:17:50] 10DBA, 10Operations, 10ops-eqiad: db1100 crashed - https://phabricator.wikimedia.org/T175973#3616923 (10jcrespo) The error on the description was on the lifecycle log. It gave the same description that you googled. [08:01:32] 10DBA, 10Data-Services: Significant replication lag for the s1, s2, and s4 wikis on labsdb100[13] - https://phabricator.wikimedia.org/T175487#3616955 (10jcrespo) 05Open>03Resolved a:03jcrespo [09:51:55] 10DBA, 10Patch-For-Review: Productionize 11 new eqiad database servers - https://phabricator.wikimedia.org/T172679#3617146 (10jcrespo) [14:15:00] 10DBA, 10Operations: decommission db1018 - https://phabricator.wikimedia.org/T176215#3617573 (10jcrespo) [14:15:04] 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, 10TCB-Team, and 7 others: Allow setting the watchlist table to read-only on a per-wiki basis - https://phabricator.wikimedia.org/T160062#3617587 (10WMDE-Fisch) [14:22:00] 10DBA, 10Operations, 10ops-eqiad: db1100 crashed - https://phabricator.wikimedia.org/T175973#3617612 (10Cmjohnson) @jcrespo pulled power and reset....power on at your convenience. [14:40:27] 10DBA, 10Operations, 10ops-eqiad: db1100 crashed - https://phabricator.wikimedia.org/T175973#3617700 (10jcrespo) Will do! Thanks. Please give me a heads up if any maintenance happens here, unless you tell me otherwise, I will put it back into production. We can put it down at any time later, but I do not wan... [15:15:49] 10DBA, 10Operations, 10ops-eqiad: db1100 crashed - https://phabricator.wikimedia.org/T175973#3617768 (10Cmjohnson) No, go ahead and put back into production. [15:48:25] 10DBA, 10Operations, 10Patch-For-Review: Firewall configurations for database hosts - https://phabricator.wikimedia.org/T104699#3617885 (10jcrespo) I think after the above patch, only the proxies are missing? [17:14:21] 10DBA, 10Patch-For-Review: Decommission db2010 and move m1 codfw to db2078 - https://phabricator.wikimedia.org/T175685#3600184 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by jynus on neodymium.eqiad.wmnet for hosts: ``` ['db2078.codfw.wmnet'] ``` The log can be found in `/var/log/wmf-auto-reima... [17:26:13] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2010 - https://phabricator.wikimedia.org/T175228#3618357 (10jcrespo) 05Open>03Resolved Let's consider this fixed and lets focus on T175685. [17:28:13] 10DBA, 10Patch-For-Review: Productionize 22 new codfw database servers - https://phabricator.wikimedia.org/T170662#3618365 (10jcrespo) [17:35:03] 10DBA, 10Patch-For-Review: Decommission db2010 and move m1 codfw to db2078 - https://phabricator.wikimedia.org/T175685#3618407 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db2078.codfw.wmnet'] ``` and were **ALL** successful. [18:01:53] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10Patch-For-Review, and 2 others: Usage tracking: record which statement group is used - https://phabricator.wikimedia.org/T151717#3618527 (10hoo) I've just scheduled the deployment to elwiki for 12:00 UTC tomorrow (September 20). [18:26:25] 10DBA: Decommission database hosts < db2030 (tracking) - https://phabricator.wikimedia.org/T176243#3618669 (10jcrespo) [18:26:47] 10DBA: Decommission database hosts < db2030 (tracking) - https://phabricator.wikimedia.org/T176243#3618684 (10jcrespo) [18:27:27] 10DBA: Decommission database hosts < db2030 (tracking) - https://phabricator.wikimedia.org/T176243#3618669 (10jcrespo) Codfw version of T134476 [22:10:02] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata: Provision a separate DB shard for wbc_entity_usage - https://phabricator.wikimedia.org/T176277#3619687 (10hoo) [22:16:24] 10DBA, 10Wikidata: Provision a separate DB shard for wbc_entity_usage - https://phabricator.wikimedia.org/T176277#3619755 (10hoo) [22:20:41] 10DBA, 10Wikidata: Provision a separate DB shard for wbc_entity_usage - https://phabricator.wikimedia.org/T176277#3619796 (10hoo) Additional question: (How) Would this work for labs? [23:58:05] 10DBA, 10Community-Tech-Sprint, 10MW-1.30-release-notes (WMF-deploy-2017-09-19 (1.30.0-wmf.19)), 10MW-1.31-release-notes (WMF-deploy-2017-09-26 (1.31.0-wmf.1)), 10Patch-For-Review: Issue with maintenance script: SELECTing revisions with high rev_id is pai... - https://phabricator.wikimedia.org/T175962#3620048