[05:28:31] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1065 - https://phabricator.wikimedia.org/T194698#4206146 (10Marostegui) a:03Cmjohnson Please @Cmjohnson proceed and change the disk [09:12:58] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2067 - https://phabricator.wikimedia.org/T194103#4206835 (10jcrespo) 05Resolved>03Open Potential SMART errors on that device. ``` PROBLEM - Device not healthy -SMART- on db2067 is CRITICAL: cluster=mysql device=cciss,9 instance=db2067:9100 job=node si... [09:33:29] 10DBA: Decommission db1053 - https://phabricator.wikimedia.org/T194634#4206935 (10jcrespo) [11:27:02] 10DBA, 10MediaWiki-Platform-Team, 10Multi-Content-Revisions, 10Structured-Data-Commons, and 2 others: Deploy MCR storage layer - https://phabricator.wikimedia.org/T174044#4207349 (10daniel) [11:48:27] 10DBA, 10MediaWiki-Platform-Team, 10Multi-Content-Revisions, 10Structured-Data-Commons, and 2 others: Deploy MCR storage layer - https://phabricator.wikimedia.org/T174044#4207429 (10daniel) [11:48:36] 10DBA, 10Multi-Content-Revisions, 10Epic, 10Schema-change: Once MCR is deployed, drop the rev_text_id, rev_content_model, and rev_content_format fields to be dropped from revision - https://phabricator.wikimedia.org/T184615#4207428 (10daniel) [11:49:08] 10DBA, 10Multi-Content-Revisions, 10Schema-change: Once MCR is deployed, drop the rev_text_id, rev_content_model, and rev_content_format fields to be dropped from revision - https://phabricator.wikimedia.org/T184615#3890073 (10daniel) [14:35:25] 10DBA, 10Operations: https://tendril.wikimedia.org/ IPv6 doesn't work - https://phabricator.wikimedia.org/T194766#4207806 (10BBlack) p:05Triage>03Normal [14:39:01] 10DBA, 10Operations: https://tendril.wikimedia.org/ IPv6 doesn't work - https://phabricator.wikimedia.org/T194766#4207818 (10jcrespo) @bblack do you have a pointer to puppet of other standalone service using the correct configuration? [14:49:54] 10DBA, 10Operations, 10Patch-For-Review: https://tendril.wikimedia.org/ IPv6 doesn't work - https://phabricator.wikimedia.org/T194766#4207806 (10Dzahn) @jcrespo ^ Add "interface::add_ip6_mapped { 'main': }" in the role class to apply to both hosts at once. That should be all that is needed. comparison: ~/... [15:03:19] 10DBA, 10Operations, 10Patch-For-Review: https://tendril.wikimedia.org/ IPv6 doesn't work - https://phabricator.wikimedia.org/T194766#4207932 (10jcrespo) a:03BBlack Please, recheck. [15:03:52] 10DBA, 10Operations, 10Patch-For-Review: https://tendril.wikimedia.org/ IPv6 doesn't work - https://phabricator.wikimedia.org/T194766#4207940 (10jcrespo) @Dzhan, thanks for the patch. [15:19:48] 10DBA, 10Operations: https://tendril.wikimedia.org/ IPv6 doesn't work - https://phabricator.wikimedia.org/T194766#4207993 (10jcrespo) [15:39:37] 10DBA, 10Operations: https://tendril.wikimedia.org/ IPv6 doesn't work - https://phabricator.wikimedia.org/T194766#4208076 (10BBlack) 05Open>03Resolved Works now, thanks! [15:45:59] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1065 - https://phabricator.wikimedia.org/T194698#4208115 (10Cmjohnson) The disk has been replaced. Please resolve once rebuild is complete [15:54:54] 10DBA, 10Patch-For-Review: Decommission db1053 - https://phabricator.wikimedia.org/T194634#4208172 (10jcrespo) Grants seem fixed. I am not moving the following ones, as they may be unused: * fabmigrate * bzmigrate * rtmigrate [17:06:59] 10DBA, 10Patch-For-Review: Decommission db1053 - https://phabricator.wikimedia.org/T194634#4208424 (10jcrespo) [17:08:30] 10DBA, 10Patch-For-Review: Decommission db1053 - https://phabricator.wikimedia.org/T194634#4204024 (10jcrespo) @mmodell upcoming failover of Phabricator database, heads up (no action needed from you). [17:21:33] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1065 - https://phabricator.wikimedia.org/T194698#4208449 (10jcrespo) 05Open>03Resolved ``` $ megacli -PDList -aALL | grep 'state' Firmware state: Online, Spun Up Firmware state: Online, Spun Up Firmware state: Online, Spun Up Firmware state: Online, Sp... [19:09:49] 10DBA, 10MediaWiki-Platform-Team, 10Multi-Content-Revisions, 10Structured-Data-Commons, and 2 others: Deploy MCR storage layer - https://phabricator.wikimedia.org/T174044#4208740 (10daniel)