[11:29:41] DBA, Wikidata, Patch-For-Review: Wikidata master database connection issue - https://phabricator.wikimedia.org/T136598#2344464 (hoo) a:hoo I looked into this very briefly and found that we unnecessarily open these connections in most job runs. My patch should fix that. [11:34:58] DBA, Operations: Install, configure and provision recently arrived db core machines - https://phabricator.wikimedia.org/T133398#2344473 (jcrespo) I've started installation, reporting issues on T135253. [11:35:38] DBA, Operations: Install, configure and provision recently arrived db core machines - https://phabricator.wikimedia.org/T133398#2344483 (jcrespo) [12:46:26] DBA, Wikimedia-General-or-Unknown, MW-1.28-release-notes, Patch-For-Review, WMF-deploy-2016-06-07_(1.28.0-wmf.5): Entries in revision table where rev_len is NULL - https://phabricator.wikimedia.org/T135414#2344630 (matmarex) @Volans @jcrespo How do you feel about running `populateRevisionLeng... [14:23:57] DBA, Wikidata, Patch-For-Review: Wikidata master database connection issue - https://phabricator.wikimedia.org/T136598#2344824 (JanZerebecki) Next one might want to look at the graph of established connections and connection attempts to the wikidatawiki master. Perhaps compare with the connection lim... [14:28:03] DBA, Wikidata, Patch-For-Review: Wikidata master database connection issue - https://phabricator.wikimedia.org/T136598#2344833 (jcrespo) @JanZerebecki - our max_connections is very high: 10000 simultaneous connections- because we use connection pooling at server side. We almost never reach that- medi... [14:50:42] DBA, Wikimedia-General-or-Unknown, MW-1.28-release-notes, Patch-For-Review, WMF-deploy-2016-06-07_(1.28.0-wmf.5): Entries in revision table where rev_len is NULL - https://phabricator.wikimedia.org/T135414#2344862 (jcrespo) I recommend a lecture of T135278, which is very different from this,... [15:26:35] DBA, MediaWiki-General-or-Unknown: Database error when filtering page log - https://phabricator.wikimedia.org/T136687#2344989 (Aklapper) [15:39:47] DBA, MediaWiki-General-or-Unknown: Database error when filtering page log - https://phabricator.wikimedia.org/T136687#2344191 (jcrespo) This is a horrible, horrible query to do: ``` SELECT log_id,log_type,log_action,log_timestamp,log_user,log_user_text,log_namespace,log_title,log_comment,log_params,log_... [16:38:50] DBA, Operations, ops-codfw: es2017 and es2019 crashed with no logs - https://phabricator.wikimedia.org/T130702#2345286 (Papaul) Yes i will need downtime for that but let me download the firmware first and i will let you or Volans know thanks. [16:57:35] DBA, Operations, ops-codfw: es2017 and es2019 crashed with no logs - https://phabricator.wikimedia.org/T130702#2345391 (jcrespo) Probably just me (Volans is away). We may want to do it on all ES servers in the end- but there is no hurry for the other hosts. Ping me at almost any time when you see me... [17:22:59] DBA, Operations, ops-codfw: db2034 degraded RAID - https://phabricator.wikimedia.org/T136583#2345518 (Papaul) The disk that was sent to me is the wrong size , so I am shipping it back. [17:23:21] DBA, Operations, ops-codfw: db2034 degraded RAID - https://phabricator.wikimedia.org/T136583#2345519 (jcrespo) :-( [18:00:18] DBA, Operations: Install, configure and provision recently arrived db core machines - https://phabricator.wikimedia.org/T133398#2345618 (jcrespo) OS installed and initial puppetization/salt of: db1079 db1080 db1081 db1083 db1084 db1087 db1088 db1090 db1091 db1092 db1093 Aside from the other hosts, they... [18:38:40] DBA, Operations, Patch-For-Review: Install, configure and provision recently arrived db core machines - https://phabricator.wikimedia.org/T133398#2345865 (Volans) @jcrespo the above CR is according to the plan I've put in T111992, of course review the plan for possible mistakes and feel free to disca... [20:48:32] DBA, Labs, Operations: disk failure on labsdb1002 - https://phabricator.wikimedia.org/T126946#2346387 (russblau) Sorry; it appears that I must have stopped reading before the end of the sentence. So, if importing revision will take roughly a month, that means that pagelinks will take another //thre... [21:14:04] DBA, Analytics, Editing-Analysis, Patch-For-Review: Reportupdater does not commit changes after each query - https://phabricator.wikimedia.org/T134950#2346574 (Milimetric) Resolved>Open Since this was merged, all reportupdater jobs seem to fail with an error like: ```ERROR - Report "cont... [21:14:31] DBA, Analytics-Kanban, Editing-Analysis, Patch-For-Review: Reportupdater does not commit changes after each query - https://phabricator.wikimedia.org/T134950#2346579 (Milimetric) a:Neil_P._Quinn_WMF>Milimetric [21:41:09] DBA, Analytics-Kanban, Editing-Analysis, Patch-For-Review: Reportupdater does not commit changes after each query - https://phabricator.wikimedia.org/T134950#2346819 (Milimetric) Yep, we have MySQL_python==1.2.3 on stat1003 and we need 1.2.5 (confirmed on my local that we get the problem with 1.2... [21:59:34] DBA, MediaWiki-Database, Schema-change: Some tables lack unique or primary keys, may allow confusing duplicate data - https://phabricator.wikimedia.org/T17441#2346972 (MBinder_WMF)