[04:55:10] 10DBA, 10Cloud-Services, 10Operations, 10Patch-For-Review: m5-master overloaded by idle connections to the nova database - https://phabricator.wikimedia.org/T188589 (10Marostegui) nova is now using 107 connections. nova_api is using 5 connections. The general health of the connection pool is a lot better w... [06:05:04] 10DBA, 10JADE, 10Operations, 10TechCom-RFC, 10Scoring-platform-team (Current): Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10daniel) > Delete the pages and drop the namespace. Note that storage isn't reclaimed, but this should be... [06:23:01] 10DBA, 10Operations, 10monitoring, 10Patch-For-Review: HAproxy on dbproxy hosts lack enough logging - https://phabricator.wikimedia.org/T201021 (10Marostegui) After all the tests, these are the options needed on `db-master.cfg`: ``` option tcplog option log-health-checks log /dev/log local0 in... [07:20:40] 10DBA, 10JADE, 10Operations, 10TechCom-RFC, 10Scoring-platform-team (Current): Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10jcrespo) > dropping namespaces is not officially supported by MediaWiki at all Also deleting data such a... [08:00:26] | [08:00:26] | .---------. [08:00:26] | /:::::::::::\ [08:00:27] | |:::::::::::::| [08:00:28] | |:::::::::::::| [08:00:29] | |::::::::::::/ [08:00:32] | |:___________\ [08:00:35] | //c \___/ /__) [08:00:37] | /' . .,_ | | -- SLICK P-B-D [08:00:40] | |': ; / \ /_/ THE GHETTO V-I-P [08:11:29] do we go fo es1011 switchover tomorrow? [08:12:30] sure [08:12:37] (I forgot about it!) [08:13:55] I will prepare it then [08:14:02] great - thank you! [08:19:25] 10DBA, 10Operations, 10monitoring, 10Patch-For-Review: HAproxy on dbproxy hosts lack enough logging - https://phabricator.wikimedia.org/T201021 (10Marostegui) 05Open>03Resolved a:03Marostegui Deployed, reloaded haproxies everywhere and I can see now the checks on the logs. [08:34:38] 10DBA, 10MediaWiki-Database, 10Patch-For-Review: Drop blob_tracking and blob_orphans everywhere - https://phabricator.wikimedia.org/T59186 (10Marostegui) a:03Marostegui I am going to start getting rid of these tables [08:38:04] 10DBA, 10MediaWiki-Database, 10Patch-For-Review: Drop blob_tracking and blob_orphans everywhere - https://phabricator.wikimedia.org/T59186 (10Marostegui) For now and to really make sure nothing uses it, I have renamed then on db1089 and will give them 24h: ``` root@db1089.eqiad.wmnet[enwiki]> show tables lik... [08:54:22] 10DBA: Switchover es2 master (es1011) to es1015 - https://phabricator.wikimedia.org/T202364 (10jcrespo) [08:54:37] 10DBA: Switchover es2 master (es1011) to es1015 - https://phabricator.wikimedia.org/T202364 (10jcrespo) a:03jcrespo [08:56:39] ^based on row C > row B, even if both masters will be on the same [08:58:44] Yeah, row B is in a worse state [08:58:57] At leat they are on separated racks [08:59:01] *least [09:27:53] 10DBA: Productionize dbproxy101[2-7].eqiad.wmnet - https://phabricator.wikimedia.org/T202367 (10Marostegui) p:05Triage>03Normal [09:28:53] 10DBA, 10Operations: rack/setup/install dbproxy101[2-7].eqiad.wmnet - https://phabricator.wikimedia.org/T196690 (10Marostegui) 05Open>03Resolved [09:52:14] do you want me to take over the x1 fixing data drifts? [09:52:26] Happy to do it as I did it lately for change_tag [10:05:11] in theory the checks should be ongoing on neodymium [10:05:20] check them on a screen called compare_x1 [10:05:37] I would prefer you to check the planned procedure for the swichover, though [10:05:40] as it is more urgent [10:06:00] I think it is ready now [10:10:31] also: https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180822T0600 [10:23:33] yeah, I was not going to do the fixes now :) [10:55:01] I have fixed all the grant issues on es1015 [10:55:40] And fixed all the password in the old format [11:06:20] will you upgrade es1015 to 10.1.35? [11:42:43] yeah, I was thinking of doing some maintenance now [11:43:45] cool! [12:55:28] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: db2069 storage crash - https://phabricator.wikimedia.org/T201603 (10Marostegui) a:03Marostegui I will get those differences fixed [13:00:09] so ok with the plan I started? [13:00:24] regarding es1015 [13:00:28] yeah, it looks good [14:14:35] volans etc.: db1052 looks decomissioned, but is still in puppet? [14:15:00] * volans checking [14:15:50] paravoid: not in puppetdb, not in debmonitor [14:16:21] it's in servermon... :/ [14:16:27] gah, tech debt [14:18:48] some of the veterans may help me with this [14:19:05] I believe modules/admin/files/enforce-users-groups.sh to no longer be necessary for the mysql user [14:19:25] if I check and all mysql users are system users, I can delete it from there, right? [14:26:46] volans: can you clean it up from servermon? (have to run to a meeting) [14:29:15] paravoid: I can look at it, never removed hosts from there :) [14:29:22] heh [14:29:53] I think one of the puppet commands should clean it up as well? [14:33:19] retries a node deactivate and node clean, nothing changed, but IIRC there was a cron, I'll have a look [14:39:02] no. deactive/clean are not enough, there's also an additional expiry logic inside servermon, IIRC they're up after ten days or so? [14:39:24] ^ akosiaris would know [14:40:42] * akosiaris in an interview [14:40:42] yeah I was about to ping him, as teh crontab runs every hour but apparently doesn't take care of cleanup [14:41:49] I remember having a conversation about this long time ago, but cannot remember the conclusion of it [14:42:39] actually I was waiting feedback on T198939 to decom it [14:42:40] T198939: Decommission servermon - https://phabricator.wikimedia.org/T198939 [15:29:01] so [15:29:03] https://github.com/servermon/servermon/commit/f09a77746637383414809e09052ec5b9c52a1efc [15:29:13] 10 days until a host is cleaned up from servermon [15:29:34] simply put, puppet commands have nothing to do with servermon [15:30:12] and adding a command in the decommisioning process for a deprecated software seemed like a bad idea [17:05:23] 10DBA, 10Operations, 10decommission, 10ops-eqiad, 10Patch-For-Review: Decommission db1052 - https://phabricator.wikimedia.org/T199861 (10Cmjohnson) [17:05:41] 10DBA, 10Patch-For-Review: Decommission db1051-db1060 (DBA tracking) - https://phabricator.wikimedia.org/T186320 (10Cmjohnson) [17:05:46] 10DBA, 10Operations, 10decommission, 10ops-eqiad, 10Patch-For-Review: Decommission db1052 - https://phabricator.wikimedia.org/T199861 (10Cmjohnson) 05Open>03Resolved [17:37:51] 10DBA, 10Patch-For-Review: Decommission db1051-db1060 (DBA tracking) - https://phabricator.wikimedia.org/T186320 (10Marostegui) 05Open>03Resolved All these hosts have now been fully decommissioned Thanks @robh a @Cmjohnson for all the help! [22:54:54] 10DBA, 10JADE, 10Operations, 10TechCom-RFC, 10Scoring-platform-team (Current): Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) >>! In T200297#4517617, @daniel wrote: >> Delete the pages and drop the namespace. Note that stor... [23:26:21] 10DBA, 10JADE, 10Operations, 10TechCom-RFC, 10Scoring-platform-team (Current): Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10Harej) >>! In T200297#4517617, @daniel wrote: >> Delete the pages and drop the namespace. Note that stora...