[05:56:55] 10DBA, 10Operations, 10netops, 10ops-eqiad, 10Patch-For-Review: db1114 connection issues - https://phabricator.wikimedia.org/T191996#4132722 (10Marostegui) [05:57:14] 10DBA, 10Operations, 10netops, 10ops-eqiad, 10Patch-For-Review: db1114 connection issues - https://phabricator.wikimedia.org/T191996#4123759 (10Marostegui) p:05Triage>03High [05:58:10] 10DBA, 10Operations, 10netops, 10ops-eqiad, 10Patch-For-Review: db1114 connection issues - https://phabricator.wikimedia.org/T191996#4123759 (10Marostegui) I have written a summary of the current state of debugging on the original task description, so it is easier to read instead of going thru all the co... [06:19:00] 10DBA, 10Collaboration-Team-Triage, 10StructuredDiscussions, 10Patch-For-Review, 10Schema-change: Drop flow_subscription table - https://phabricator.wikimedia.org/T149936#4132729 (10Marostegui) @Catrope the one in officewiki (s3) is also empty. Can that one be dropped as well? [06:21:56] 10DBA, 10Collaboration-Team-Triage, 10StructuredDiscussions, 10Patch-For-Review, 10Schema-change: Drop flow_subscription table - https://phabricator.wikimedia.org/T149936#4132732 (10Marostegui) a:03Marostegui flow_subscription has been deleted from x1. It was empty, so no backup is needed. It didn't ha... [07:16:38] 10DBA, 10Operations, 10netops, 10ops-eqiad, 10Patch-For-Review: db1114 connection issues - https://phabricator.wikimedia.org/T191996#4132767 (10Marostegui) [07:35:05] 10DBA, 10Operations, 10netops, 10ops-eqiad, 10Patch-For-Review: db1114 connection issues - https://phabricator.wikimedia.org/T191996#4132775 (10Marostegui) [07:46:49] 10DBA, 10Operations, 10netops, 10ops-eqiad, 10Patch-For-Review: db1114 connection issues - https://phabricator.wikimedia.org/T191996#4132785 (10Marostegui) [13:00:57] https://grafana.wikimedia.org/dashboard/db/mysql?panelId=11&fullscreen&orgId=1&var-dc=codfw%20prometheus%2Fops&var-server=dbstore2001&var-port=13318&from=1523873460486&to=1523883644492 [13:01:07] ^the purge threads seem to start kicking in [13:02:36] but lag seems to be getting worse [13:11:06] I guess it is still cold [13:12:14] probably, + extra load [13:19:40] 10DBA, 10Operations, 10netops, 10ops-eqiad, 10Patch-For-Review: db1114 connection issues - https://phabricator.wikimedia.org/T191996#4133478 (10Marostegui) After the last reboot the errors have moved from being at times like: XX:10:11 XX:20:11 XX:30:11 To: XX:04:11 XX:24:11 XX:34:11 [13:22:47] 10DBA, 10Operations, 10netops, 10ops-eqiad, 10Patch-For-Review: db1114 connection issues - https://phabricator.wikimedia.org/T191996#4133504 (10Marostegui) [13:40:35] 10DBA, 10Wikidata: Optimize recentchanges and wbc_entity_usage table across wikis - https://phabricator.wikimedia.org/T187521#4133553 (10jcrespo) commonswiki errors due to deadlocks on `INSERT IGNORE INTO wbc_entity_usage ` seem to be common (not too worrying, but on of the most comon database errors), could t... [13:41:07] 10DBA, 10Operations, 10netops, 10ops-eqiad, 10Patch-For-Review: db1114 connection issues - https://phabricator.wikimedia.org/T191996#4133555 (10Marostegui) As another test to discard issues - I have rebooted db1114 with an older kernel. So it is now running ``` root@db1114:~# uname -a Linux db1114 4.9.0... [16:11:32] 10DBA, 10Operations, 10netops, 10ops-eqiad, 10Patch-For-Review: db1114 connection issues - https://phabricator.wikimedia.org/T191996#4134090 (10Marostegui)