[01:56:48] DBA, Labs: High replag on db1069 - https://phabricator.wikimedia.org/T136789#2347971 (yuvipanda) [02:04:35] DBA, Labs: High replag on db1069 - https://phabricator.wikimedia.org/T136789#2347975 (yuvipanda) People with access can keep a look on the db1069 replag at https://tendril.wikimedia.org/tree [02:12:22] DBA, Labs: High replag on db1069 - https://phabricator.wikimedia.org/T136789#2347978 (yuvipanda) p:Triage>High [06:42:23] DBA, Labs, Operations: disk failure on labsdb1002 - https://phabricator.wikimedia.org/T126946#2348194 (jcrespo) Not necessarily, it is probably a less wide table and will not have so many metadata issues as revision. [07:13:17] DBA, Operations, Patch-For-Review: Install, configure and provision recently arrived db core machines - https://phabricator.wikimedia.org/T133398#2348216 (jcrespo) @Volans, while I appreciate the patch, I cannot use it as-is, as many machines have not yet been properly installed due to network/config... [07:53:49] DBA, Labs: High replag on db1069 - https://phabricator.wikimedia.org/T136789#2348243 (yuvipanda) db1069 has no replag now, and replag on labsdb1001 is also reducing. [08:16:31] DBA, Labs: High replag on db1069 - https://phabricator.wikimedia.org/T136789#2348276 (yuvipanda) Open>Resolved a:yuvipanda All clear. [09:58:58] DBA, Operations, Patch-For-Review: Investigate/decom db2001-db2009 - https://phabricator.wikimedia.org/T125827#1998016 (MoritzMuehlenhoff) I noticed this ticket when checking for db servers without base::firewall enabled: Summarising: - db2008/db2009 were removed from mediawiki in https://gerrit.wiki... [10:08:01] DBA, Operations, Patch-For-Review: Investigate/decom db2001-db2009 - https://phabricator.wikimedia.org/T125827#2348590 (jcrespo) I would remove them all when Daniel finishes his work. [10:08:53] DBA, Operations, Patch-For-Review: Investigate/decom db2001-db2009 - https://phabricator.wikimedia.org/T125827#2348591 (jcrespo) [10:09:15] DBA, Operations: Investigate/decom db2001-db2009 - https://phabricator.wikimedia.org/T125827#1998016 (jcrespo) [12:42:04] DBA, Wikidata, Patch-For-Review, Wikidata-Sprint-2016-05-24: Wikidata master database connection issue - https://phabricator.wikimedia.org/T136598#2348958 (daniel) p:Triage>High [12:45:10] DBA, Wikidata, Patch-For-Review, Wikidata-Sprint-2016-05-24: Wikidata master database connection issue - https://phabricator.wikimedia.org/T136598#2348963 (daniel) Taking this into the sprint because it's an ongoing performance issue and is potentially causing data loss. Marius already made a fix. [13:01:08] DBA, Wikidata, Patch-For-Review, Wikidata-Sprint-2016-05-24: Wikidata master database connection issue - https://phabricator.wikimedia.org/T136598#2348986 (daniel) FTR, it remains unclear why the problem started when it started. The code did not change at that point, afaik. Maybe we just crossed... [15:55:38] DBA, Operations, Patch-For-Review: Install, configure and provision recently arrived db core machines - https://phabricator.wikimedia.org/T133398#2349511 (jcrespo) This is the last updated list: db1079 db1080 db1081 db1083 db1084 db1087 db1088 db1089 db1090 db1091 db1092 db1093 db1094 [16:27:18] DBA, Wikidata, Wikidata-Sprint-2016-05-24: Wikidata master database connection issue - https://phabricator.wikimedia.org/T136598#2340469 (hoo) Open>Resolved [16:27:48] DBA, Wikidata, Wikidata-Sprint-2016-05-24: Wikidata master database connection issue - https://phabricator.wikimedia.org/T136598#2340469 (hoo) Change has been merged on master, can be backported, if needed. [18:19:08] DBA, Operations, ops-codfw: db2034 degraded RAID - https://phabricator.wikimedia.org/T136583#2350259 (Papaul) a:Papaul>jcrespo I received a 2.5" SAS 600GB 15K from HP because they do not have any 3.5" disks . So db2034 will have 11x3.5" SAS 600GB 15K and 1x2.5" 600GB 15k. I don't think this w... [18:20:42] DBA, Operations, Patch-For-Review: Investigate/decom db2001-db2009 - https://phabricator.wikimedia.org/T125827#2350265 (Dzahn) 11:23 < mutante> !log db2007 shutdown, schedule eternal downtime 11:24 < mutante> !log db2007, revoke puppet cert, delete salt key, nuke from stored configs / icinga [18:25:36] DBA, Operations, hardware-requests, ops-codfw, Patch-For-Review: Decommission es2005-es2010 - https://phabricator.wikimedia.org/T134755#2350326 (Papaul) [21:29:19] DBA, Beta-Cluster-Infrastructure, Flow, Collab-Team-2016-Apr-Jun-Q4: Run Flow External Store migration in dry-run mode on Beta - https://phabricator.wikimedia.org/T119567#2351104 (Mattflaschen-WMF)