[15:01:11] 10DBA: db1115 (tendril) crashed twice in 24h - https://phabricator.wikimedia.org/T231165 (10Marostegui) [15:02:41] 10DBA: db1115 (tendril) crashed twice in 24h - https://phabricator.wikimedia.org/T231165 (10Marostegui) On the first crash MySQL never went down. On the second crash, it was an OOM: ` [12466353.895744] mysqld invoked oom-killer: gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0-1, order=0, oom_scor... [15:03:59] 10DBA: db1115 (tendril) crashed twice in 24h - https://phabricator.wikimedia.org/T231165 (10Marostegui) HW logs are clean, so this looks like the "usual" memory leak from tendril. We might need to decrease the pool size even more, or start disabling non-used stuff in tendril (graphs and things like that): {F3010... [15:50:51] 10DBA: db1115 (tendril) crashed twice in 24h - https://phabricator.wikimedia.org/T231165 (10Marostegui) The first set of pages were due to the NREP daemon being killed by OOM and all the checks failing: ` Aug 25 00:03:51 db1115 systemd[1]: nagios-nrpe-server.service: Main process exited, code=exited, status=2/IN... [15:52:28] 10DBA: db1115 (tendril) paged twice in 24h due to OOM - https://phabricator.wikimedia.org/T231165 (10Marostegui) [18:26:44] 10DBA, 10GlobalBlocking, 10User-DannyS712: Alter globalblocking table to allow longer comments - https://phabricator.wikimedia.org/T231170 (10DannyS712) [18:31:09] 10DBA, 10GlobalBlocking, 10Patch-For-Review, 10User-DannyS712: Alter globalblocking table to allow longer comments - https://phabricator.wikimedia.org/T231170 (10DannyS712) [18:39:50] 10DBA, 10GlobalBlocking, 10Patch-For-Review, 10Schema-change, 10User-DannyS712: Alter globalblocking table to allow longer comments - https://phabricator.wikimedia.org/T231170 (10Reedy) [18:41:28] 10DBA, 10GlobalBlocking, 10Patch-For-Review, 10Schema-change, 10User-DannyS712: Alter globalblocking tables to allow longer comments - https://phabricator.wikimedia.org/T231170 (10Reedy) [19:31:23] 10Blocked-on-schema-change, 10GlobalBlocking: Extend gbw_reason/gb_reason on WMF production - https://phabricator.wikimedia.org/T231172 (10Reedy) [19:31:35] 10DBA, 10GlobalBlocking, 10Patch-For-Review, 10Schema-change, 10User-DannyS712: Alter globalblocking tables to allow longer comments - https://phabricator.wikimedia.org/T231170 (10Reedy) [19:31:37] 10Blocked-on-schema-change, 10GlobalBlocking: Extend gbw_reason/gb_reason on WMF production - https://phabricator.wikimedia.org/T231172 (10Reedy) [19:33:09] 10Blocked-on-schema-change, 10GlobalBlocking: Extend gbw_reason/gb_reason on WMF production - https://phabricator.wikimedia.org/T231172 (10Reedy) [19:34:28] 10Blocked-on-schema-change, 10GlobalBlocking: Extend gbw_reason/gb_reason on WMF production - https://phabricator.wikimedia.org/T231172 (10DannyS712) Thanks for following up [21:13:46] 10Blocked-on-schema-change, 10GlobalBlocking: Alter gbw_reason/gb_reason/gbw_by_text on WMF production - https://phabricator.wikimedia.org/T231172 (10DannyS712) [21:29:23] 10Blocked-on-schema-change, 10GlobalBlocking: Alter gbw_reason/gb_reason/gbw_by_text on WMF production - https://phabricator.wikimedia.org/T231172 (10Reedy) Altered `centralauth.globalblocks` on beta. `beta-update-databases-eqiad` has updated the tables on the various wikis