[07:46:13] DBA, Wikidata, Wikidata-Sprint-2016-05-24: Wikidata master database connection issue - https://phabricator.wikimedia.org/T136598#2360125 (jcrespo) Resolved>Open This issue happened again today between 4:26 and 7:20 (I may have done something to end it because I had to do some unrelated server... [09:23:07] DBA: Defragment db1070, db1082, db1087, db1092 - https://phabricator.wikimedia.org/T137191#2360296 (jcrespo) [09:23:52] DBA: Defragment db1070, db1082, db1087, db1092 - https://phabricator.wikimedia.org/T137191#2360308 (jcrespo) [09:23:53] DBA, Operations, Patch-For-Review: Install, configure and provision recently arrived db core machines - https://phabricator.wikimedia.org/T133398#2360309 (jcrespo) [09:23:56] DBA: Defragment db1070, db1082, db1087, db1092 - https://phabricator.wikimedia.org/T137191#2360296 (jcrespo) p:Triage>Low [10:08:45] DBA, Wikidata, Wikidata-Sprint-2016-05-24: Wikidata master database connection issue - https://phabricator.wikimedia.org/T136598#2360341 (hoo) Not surprising, that this is still an issue: The changes have not been backported/ deployed, yet. I can try to arrange a deploy today, if needed. To anyone... [10:11:30] DBA, Wikidata, Wikidata-Sprint-2016-05-24: Wikidata master database connection issue - https://phabricator.wikimedia.org/T136598#2360349 (jcrespo) @hoo, my apologies- I thought it had been already deployed. Let me keep it open until it deploys so that 3rd parties can see it can happen for now- you ca... [14:22:51] DBA, Notifications, Schema-change: Temporary index for Echo backfillReadBundles.php? - https://phabricator.wikimedia.org/T137100#2360947 (Catrope) >>! In T137100#2357623, @jcrespo wrote: >> The current version of the script > > I will give it a look, then get back to here with suggestions if needed... [16:47:33] DBA, Labs, Patch-For-Review: Make watchlist table available on labs - https://phabricator.wikimedia.org/T59617#2361357 (jcrespo) [16:56:11] DBA, Labs, Patch-For-Review: Make watchlist table available on labs - https://phabricator.wikimedia.org/T59617#2361443 (jcrespo) After some thinking, and my thoughts on why labs breaks so easily T136618#2356834, I think this is one of the cases that would fail to be replicated accurately- because wat... [17:14:31] DBA, Wikidata, Wikidata-Sprint-2016-05-24: Wikidata master database connection issue - https://phabricator.wikimedia.org/T136598#2361568 (JanZerebecki) ``` 19:08 < jynus> jzerebecki, as far as I can see, it is only affecting itself, and not other connections 19:09 < jynus> (the wikidata job queue exe... [18:18:29] DBA, Operations, ops-codfw: es2017 and es2019 crashed with no logs - https://phabricator.wikimedia.org/T130702#2361778 (Papaul) a:Papaul>jcrespo Update note on both systems BIOS 1.5.4 to 2.0.2 IDRAC 2.21 to 2.30 Dell uEFI diagnostics Dell Os Driver Pack 15.10 to 16.03 PERC H730 Controller... [18:31:02] DBA, Operations, ops-codfw: db2034 degraded RAID - https://phabricator.wikimedia.org/T136583#2361800 (Papaul) @jcrespo can you please attach the log? [19:42:01] DBA, Operations, ops-codfw: es2017 and es2019 crashed with no logs - https://phabricator.wikimedia.org/T130702#2362003 (jcrespo) A bunch of errors is making netfilter and ntp fail on es2017. On the admin console: ``` MEM0701: Correctable memory error rate exceeded for DIMM_A2. 2016-06-07T15:28:36-0... [19:42:49] DBA, Operations, ops-codfw: es2017 and es2019 crashed with no logs - https://phabricator.wikimedia.org/T130702#2143716 (jcrespo) Resolved>Open [19:53:03] DBA, Operations, ops-codfw: es2017 and es2019 crashed with no logs - https://phabricator.wikimedia.org/T130702#2362042 (jcrespo) restarting es2017 fixed the software issues, but this is clearly not in a closed state. This is not the highest priority, but clearly there is a hardware defect here (board?). [19:57:41] DBA, Operations, ops-codfw: db2034 degraded RAID - https://phabricator.wikimedia.org/T136583#2362064 (jcrespo) Sorry about that, log was obtained by @robh and was pasted here: {P3211}