[06:07:03] 10DBA, 10Data-Services: Prepare and check storage layer for taywiki - https://phabricator.wikimedia.org/T275836 (10Marostegui) We received a bunch of alerts for this wiki PII on wiki replicas. Even though no data was leaked, as there are no views...looks like the procedure for creating new wikis wasn't followe... [06:23:24] 10DBA, 10Data-Services: Prepare and check storage layer for mnwwiktionary - https://phabricator.wikimedia.org/T276126 (10Marostegui) This wiki was created but this task wasn't updated - I have sanitized it after receiving the private data alert. [06:30:21] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for mnwwiktionary - https://phabricator.wikimedia.org/T276126 (10Marostegui) `_p` databases and grants added. This is ready for views creation on labsdb1009, 1010, 1011 and clouddb1016, 1020, 1021 [06:37:48] 10DBA, 10decommission-hardware, 10Patch-For-Review: decommission db1084.eqiad.wmnet - https://phabricator.wikimedia.org/T276302 (10Marostegui) [06:52:59] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) [06:53:09] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) db1161 is now being pooled [06:57:47] 10DBA, 10Data-Services: Prepare and check storage layer for taywiki - https://phabricator.wikimedia.org/T275836 (10Marostegui) I have sanitized this data, is this wiki fully created? Can I send this for views creation? [07:12:25] 10DBA, 10SRE, 10Wikimedia-Mailing-lists, 10Patch-For-Review: Create test databases for mailman3 - https://phabricator.wikimedia.org/T256538 (10Marostegui) Grants added: ` root@db1128.eqiad.wmnet[(none)]> show grants for 'testmailman3'@'208.80.154.13'; +----------------------------------------------------... [07:15:12] 10DBA, 10SRE, 10Wikimedia-Mailing-lists, 10Patch-For-Review: Create test databases for mailman3 - https://phabricator.wikimedia.org/T256538 (10Ladsgroup) Yup, the ferm is missing (T277286) I'll add it. [07:20:50] good morning folks [07:20:59] I am checking dbstore1004, again memory alert :( [07:26:09] elukey: maybe we can decrease a bit its buffer pools [07:26:27] 10DBA, 10SRE, 10Wikimedia-Mailing-lists, 10Patch-For-Review: Create test databases for mailman3 - https://phabricator.wikimedia.org/T256538 (10Marostegui) 05Open→03Resolved Excellent, if there's a tracking task for that missing bit, I am going to close this as done. Please re-open if you find issues wi... [07:26:38] marostegui: yeah I was filing a change for that [07:34:11] 10DBA, 10Data-Services, 10MW-1.36-notes (1.36.0-wmf.34; 2021-03-09), 10Patch-For-Review, and 3 others: Create Wikipedia Kari Seediq - https://phabricator.wikimedia.org/T276246 (10Marostegui) `trvwiki_p` database and grants added. #cloud-services-team this is ready for views creation on labsdb1009, 1010, 1... [07:59:11] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['db1158.eqiad.wmnet'] ` The log ca... [08:00:51] 10DBA, 10Analytics-Radar, 10Patch-For-Review: mariadb on dbstore hosts, and specifically dbstore1004, possible memory leaking - https://phabricator.wikimedia.org/T270112 (10elukey) Restarted all instances on dbstore1004 today :( [08:02:00] 10DBA, 10Data-Services: Prepare and check storage layer for taywiki - https://phabricator.wikimedia.org/T275836 (10Urbanecm) >>! In T275836#6933309, @Marostegui wrote: > I have sanitized this data, is this wiki fully created? Can I send this for views creation? Sorry about that, I forgot to update this task :... [08:08:00] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for taywiki - https://phabricator.wikimedia.org/T275836 (10Marostegui) Thanks @Urbanecm `taywiki_p` database and grants added. #cloud-services-team this is ready for views creation on labsdb1009, 1010, 1011 and clouddb1... [08:10:38] 10DBA, 10Data-Services, 10MW-1.36-notes (1.36.0-wmf.34; 2021-03-09), 10Patch-For-Review, and 3 others: Create Wikipedia Kari Seediq - https://phabricator.wikimedia.org/T276246 (10Marostegui) 05Resolved→03Open Re-opening so #wmcs-kanban can create the views. [08:20:23] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db1158.eqiad.wmnet'] ` and were **ALL** successful. [10:23:52] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) db1165 is now replicating, I am checking all the tables now. [11:31:06] marostegui: hey, is there a difference between "a lot of small insert queries for each item in a batch", and "one insert query per a batch, adding all needed rows" in a maintenance script, or would one of the options be more preferable? [11:31:24] Urbanecm: meeting, will get back to you once done! [11:31:30] sure, thanks! [12:04:01] * Reedy waits for a "it depends" answer [12:04:03] * Reedy coughs [12:04:05] * Reedy smiles [12:05:29] Urbanecm: it depends [12:05:34] XD [12:05:41] i kinda expected this :D [12:05:46] depends on...what? :D [12:05:49] "what does it depend on?" [12:05:53] Urbanecm: Nah, we normally prefer smaller but fast inserts [12:05:58] * Urbanecm hopes the answer is not "marostegui's mood" [12:06:06] Urbanecm: and if possible with wait for replication between them or "some of them" [12:06:39] Majavah: we can cope with lots of fast operations, but big ones may generate lag, locking etc... [12:06:46] so sth like small & fast inserts and a wait for replication after each 200 would be good? [12:06:54] s/good/db friendly/ [12:07:14] Urbanecm: Sure, that can be tried, we can lower/increase the batch size if needed after seeing how the db performs [12:07:37] it's an one-off script [12:07:52] ah ok :) [12:07:57] with wait for replication we should be fine [12:08:33] thanks :) [12:08:40] yw! [14:54:40] FYI I wasn't able to open dbtree.w.o: database connection to tendril on tendril-backend.eqiad.wmnetfailed [14:57:02] volans: yeah, we are testnig the migration of dbmonitor to buster [14:57:06] moritzm and I are on it [14:57:17] this is probably some ongoing work manuel or moritzm is doing and IPv6 issues: https://wikitech.wikimedia.org/wiki/MariaDB#IPv6_and_MariaDB [14:57:20] ok, as far as it's known, all good :) [14:57:45] "Address is not a valid IPv4 or IPv6 address" [14:58:50] could be a missleading error [15:33:37] dbtree is back [16:56:54] hello folks, one question - what is the best way to change the max_user_connections for our user on clouddb1021 ? [16:57:06] I think that we applied a special override on labsdb1012 [16:57:33] elukey: yeah, I recall that [16:57:36] what's the user? [16:57:43] s53272 [16:57:55] Do you remember how many it had? [16:58:59] That user wasn't tracked on the grants for wikireplicas, so I cannot see how many we gave it [16:59:29] marostegui: I think that we can start with 50 ? [16:59:31] sure [17:00:26] elukey: done, applied to all the sections [17:00:33] I will add it to puppet tomorrow, as I need to leave now [17:01:23] marostegui: ahahhah faster than light, thanks! [17:02:37] <3