[07:59:23] 10DBA, 10Core Platform Team, 10Schema-change, 10User-DannyS712: iwlinks indexes should be UNIQUE INDEXes - https://phabricator.wikimedia.org/T256842 (10DannyS712) [08:47:50] working on fixing x1 dumps- they broke because documented grant issue on upgrade [08:55:07] wee. with my latest 2 CRs merged, we have some pretty nice aliases for cumin: `cumin "A:db-section-x1 and A:db-role-master"` [08:57:08] wait, didn't those existed already? [08:57:31] I remember something existed based on prometheus/monitoring parameters [08:57:49] I am getting "The MariaDB server is running with the --read-only option so it cannot execute this statement" error on bacula [08:58:13] ah, I think that was yesterday's crash [09:00:40] kormat: your task should be now to check that zarcillo, tendril and cumin agree on those roles/sections and setup some monitoring 0:-D [09:01:10] jynus: i think you're thinking of https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/570161, which didn't get merged [09:01:19] that's an excellent point. i'll create a task. [09:01:30] I see [09:01:38] so there was the idea of it, but never get done [09:01:44] *got [09:02:11] I think I may have moved prometheus to zarcillo [09:06:20] 10DBA, 10Operations, 10User-Kormat: Add monitoring to ensure that puppet/tendril/zarcillo all agree on the set of sections that exist - https://phabricator.wikimedia.org/T256845 (10Kormat) [09:06:29] 10DBA, 10Operations, 10User-Kormat: Add monitoring to ensure that puppet/tendril/zarcillo all agree on the set of sections that exist - https://phabricator.wikimedia.org/T256845 (10Kormat) p:05Triage→03Medium [09:08:53] 10DBA, 10Puppet, 10cloud-services-team (Kanban): labtestpuppetmaster2001 is failing to backup - https://phabricator.wikimedia.org/T256846 (10jcrespo) [09:09:16] ^I have filed a task for cloud to research failing backups [09:19:16] s4 grew again a 10% after compression week-to-week [09:21:21] if it follow the same growth data size will double in 7 weeks [09:21:25] *follows [09:22:06] ouch [09:24:10] s4 was preciselly upgraded with much more size recently, but for a different reason [09:26:44] so while the growth can be seen quite steep: https://grafana.wikimedia.org/d/000000377/host-overview?panelId=28&fullscreen&orgId=1&refresh=5m&var-server=db1149&var-datasource=thanos&var-cluster=mysql&from=1591003571496&to=1593595571496 [09:27:20] server is at 17% utilization only [09:34:59] * kormat nods [10:49:23] 10DBA, 10Operations, 10CAS-SSO, 10Patch-For-Review, 10User-jbond: Request new database for idp-test.wikimedia.org - https://phabricator.wikimedia.org/T256120 (10jcrespo) a:03jbond The host has been added the missing monitoring and open ports, as well as updated on tendril and zarcillo. If this is all... [10:58:04] 10DBA, 10Operations, 10SRE-tools, 10Patch-For-Review, 10User-Kormat: Audit all cumin queries in switchdc scripts - https://phabricator.wikimedia.org/T243935 (10Kormat) 05Open→03Resolved `mysql_legacy` is now updated, so i think this can be closed. [12:09:48] 10DBA, 10Operations, 10User-Kormat: Add mysql_role and section profiles to remaining mariadb roles - https://phabricator.wikimedia.org/T256866 (10Kormat) [12:09:56] 10DBA, 10Operations, 10User-Kormat: Add mysql_role and section profiles to remaining mariadb roles - https://phabricator.wikimedia.org/T256866 (10Kormat) p:05Triage→03Medium [12:19:42] jynus: can you tell me what the FIXME here is for? https://github.com/wikimedia/puppet/blob/production/modules/profile/manifests/mariadb/misc/tendril.pp#L13 [12:31:29] 10DBA, 10Operations, 10User-Kormat: Add mysql_role and section profiles to remaining mariadb roles - https://phabricator.wikimedia.org/T256866 (10Kormat) [12:33:31] 10DBA, 10Patch-For-Review: Make checksum parallel to the data transfer in transferpy package - https://phabricator.wikimedia.org/T254979 (10Privacybatm) @jcrespo Can you please tell me why is the following not working? `python3 transferpy/transfer.py --type=xtrabackup --no-encrypt --no-checksum --no-compress... [12:40:11] 10DBA, 10Operations, 10User-Kormat: Add mysql_role and section profiles to remaining mariadb roles - https://phabricator.wikimedia.org/T256866 (10Kormat) [12:40:49] 10DBA, 10Operations, 10User-Kormat: Add mysql_role and section profiles to remaining mariadb roles - https://phabricator.wikimedia.org/T256866 (10Kormat) [12:48:10] 10DBA, 10Operations, 10User-Kormat: Add mysql_role and section profiles to remaining mariadb roles - https://phabricator.wikimedia.org/T256866 (10Kormat) [12:59:00] 10DBA, 10Patch-For-Review: Make checksum parallel to the data transfer in transferpy package - https://phabricator.wikimedia.org/T254979 (10Privacybatm) @jcrespo Can you please tell me a way to corrupt the **source socket** in `xtrabackup`. By corruption, I meant some changes, for example: In case of file I... [14:17:12] kormat: If I remember correctly, tendril was initially setup as a master-replica setup, but it didn't work because tendril [14:17:27] ah, haha [14:17:33] in that case i'll just remove the FIXME [14:17:33] so at some point it shouldn't be standalone, but 2 redundant servers [14:20:04] if the prometheus parameters are no longer being used, we can maybe removed on a further patch? [14:20:12] *remove them [14:20:26] i don't know what they do, i haven't looked. [14:20:52] huh. nothing. yeah ok :) [14:21:09] I think they where added to gather the classification the same you did [14:21:34] but later it was seen that it didn't work because instance != host configuration [14:21:44] pluse there were some hosts that didn't need prometheus bug need inventory [14:21:54] gotcha [14:22:00] so prometheus classification was moved to zarcillo [14:22:21] not related to this patch, just mentioning it as future work [14:22:45] to start removing redundancies [14:25:17] +1 [14:41:44] 10DBA, 10Operations, 10User-Kormat: Remove unused parameters from profile::mariadb::monitor::prometheus - https://phabricator.wikimedia.org/T256879 (10Kormat) [14:51:11] 10DBA, 10Patch-For-Review: Make checksum parallel to the data transfer in transferpy package - https://phabricator.wikimedia.org/T254979 (10jcrespo) > ERROR: file was not found on the target path /home/privacybatm/testing/xtrabackup_info after transfer That is how xtrabackup backups are checked- they expect a... [14:55:39] 10DBA, 10Patch-For-Review: Make checksum parallel to the data transfer in transferpy package - https://phabricator.wikimedia.org/T254979 (10jcrespo) If you shutdown mysql (sudo systemctl stop mariadb) and copy away /srv/sqldata before corrupting it, you will be able to recover a damaged mariadb data directory... [15:02:39] 10DBA, 10Gerrit, 10Patch-For-Review: Make sure both `reviewdb-test` (used forgerrit upgrade testing) and `reviewdb` (formerly production) databases get torn down - https://phabricator.wikimedia.org/T255715 (10Dzahn) @jcrespo Chris has asked to keep it for a couple more days. [15:04:51] 10DBA, 10Gerrit, 10Patch-For-Review: Make sure both `reviewdb-test` (used forgerrit upgrade testing) and `reviewdb` (formerly production) databases get torn down - https://phabricator.wikimedia.org/T255715 (10jcrespo) Perfect, no problem. [17:50:41] 10DBA, 10Performance-Team, 10Patch-For-Review: Database for XHGui profiles - https://phabricator.wikimedia.org/T254795 (10Dzahn) Also shared the password with Krinkle who added credentials to PrivateSettings.php. I think this ticket is done now.