[01:24:05] 10DBA, 10Operations, 10Wikimedia-Mailing-lists: Create databases for mailman3 - https://phabricator.wikimedia.org/T256538 (10Dzahn) Here are the largest ones as requested by Ladsgroup on T262773 ` 10G helpdesk-l 6.1G reading-web-team.mbox 6.1G arbcom-l 5.5G wmfall 4.8G reading-web-team 3.8G wikidata-bugs 3.... [05:05:13] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: dbproxy1020 PS disconnected - https://phabricator.wikimedia.org/T262998 (10Marostegui) @Cmjohnson thanks for resetting the logs yesterday. Unfortunately, the idrac still reports one of the power supplies as down. {P12612} [05:33:55] 10DBA, 10Patch-For-Review: Productionize es20[26-34] and es10[26-34] - https://phabricator.wikimedia.org/T261717 (10Marostegui) on-going transfers: es2012 -> es2030 es2018 -> es2029 [05:36:49] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review, 10User-Kormat: db2125 crashed - mgmt iface also not available - https://phabricator.wikimedia.org/T260670 (10Marostegui) @Papaul I guess we should do that...and then wait again for the next crash and start the whole loop again. @Papaul let me know whe... [05:47:00] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: Physically move db1131 from B5 to C8 - https://phabricator.wikimedia.org/T262901 (10Marostegui) @Cmjohnson mysql is down on this host. I haven't powered it off yet, just to see if you can give me the new IP before I do so, so I can change it on the host so it boo... [06:47:10] 10DBA, 10Operations, 10Wikimedia-Mailing-lists: Create databases for mailman3 - https://phabricator.wikimedia.org/T256538 (10jcrespo) > That's a pretty big database and I do share Jaime's concerns. > I would definitely not store that on m2 as it already has `OTRS` which is around 500GB already. If we had to... [06:56:16] 10DBA, 10Operations, 10Wikimedia-Mailing-lists: Create databases for mailman3 - https://phabricator.wikimedia.org/T256538 (10Marostegui) >>! In T256538#6469302, @jcrespo wrote: >> That's a pretty big database and I do share Jaime's concerns. >> I would definitely not store that on m2 as it already has `OTRS... [06:58:37] 10DBA, 10Operations, 10Wikimedia-Mailing-lists: Create databases for mailman3 - https://phabricator.wikimedia.org/T256538 (10jcrespo) Yes, agree, once wikitech is moved, we can keep it and not introduce a new section, keep it for mailman, that will need significant IOPS and disk. [09:24:46] Good morning :D I'm picking up T246415 for investigating how to regroup s8. I need some observability. I know the origin is not visible to the db (either is from en.wikipedia.org or www.wikidata.org) but it would be great if I can get some insight on tables on s8 are being read a lot. Any ideas? Maybe performance schema? (T195578) [09:24:47] T246415: Investigate a different db load groups for wikidata / wikibase - https://phabricator.wikimedia.org/T246415 [09:24:47] T195578: Deploy access to performance_schema/sys for the administrative mediawiki account (mediawiki deployers) - https://phabricator.wikimedia.org/T195578 [09:29:22] Amir1: I will get back to you later [09:30:02] Thanks! [11:03:57] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: New Date - Thur, Sept 17: PDU Upgrade 12pm-4pm UTC- Racks D1 and D2 - https://phabricator.wikimedia.org/T261459 (10Marostegui) mysql stopped on db1125 on all the instances [11:10:45] 10DBA, 10Patch-For-Review: Productionize es20[26-34] and es10[26-34] - https://phabricator.wikimedia.org/T261717 (10Marostegui) [11:11:05] 10DBA, 10Patch-For-Review: Productionize es20[26-34] and es10[26-34] - https://phabricator.wikimedia.org/T261717 (10Marostegui) es2031 is fully pooled into es2 [12:52:35] jynus: it slipped a bit, but i'm going to work on getting a release of wmfmariadbpy out soonish [12:54:41] 10DBA, 10Operations, 10Patch-For-Review, 10User-Kormat, 10User-jbond: Refactor mariadb puppet code - https://phabricator.wikimedia.org/T256972 (10Kormat) [13:10:43] 10DBA, 10Operations: Remove sections from db configs - https://phabricator.wikimedia.org/T263127 (10Kormat) [13:11:20] 10DBA, 10Operations, 10User-Kormat: Remove sections from db configs - https://phabricator.wikimedia.org/T263127 (10Kormat) p:05Triage→03Medium [13:12:16] marostegui: ^ as discussed [13:12:45] thanks, I will add some comments/concerns and subscribe some people that I would like to hear input from (MW side) [13:12:56] great, thanks [13:15:37] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review, 10User-Kormat: db2125 crashed - mgmt iface also not available - https://phabricator.wikimedia.org/T260670 (10Papaul) @Marostegui you can at anytime. Thanks [13:16:45] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review, 10User-Kormat: db2125 crashed - mgmt iface also not available - https://phabricator.wikimedia.org/T260670 (10Marostegui) @Papaul doing it now, thanks - will ping you once it is ready for you [13:17:46] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review, 10User-Kormat: db2125 crashed - mgmt iface also not available - https://phabricator.wikimedia.org/T260670 (10Marostegui) @Papaul host off - you can go ahead [13:46:45] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review, 10User-Kormat: db2125 crashed - mgmt iface also not available - https://phabricator.wikimedia.org/T260670 (10Papaul) upgrade BIOS from BIOS Version 2.8.1 to 2.8.2 changed profile settings from to Performace @Marostegui all... [13:50:36] 10DBA, 10Security-Team, 10Performance Issue, 10SecTeam Discussion, 10Security: Deploy access to performance_schema/sys for the administrative mediawiki account (mediawiki deployers) - https://phabricator.wikimedia.org/T195578 (10Ladsgroup) Can I at least have it temporarily? I need lots of information fo... [13:51:46] 10DBA, 10Operations, 10Performance-Team, 10Platform Engineering, 10User-Kormat: Remove sections from db configs - https://phabricator.wikimedia.org/T263127 (10Marostegui) So in the past, the hosts serving those 5 groups used to have different schema partitioning (different indexes, PKs and even partition... [13:52:55] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review, 10User-Kormat: db2125 crashed - mgmt iface also not available - https://phabricator.wikimedia.org/T260670 (10Marostegui) Thank you @Papaul - I will start mysql and start slowing repooling the host in production. [13:53:22] 10DBA, 10Operations, 10Performance-Team, 10Platform Engineering, 10User-Kormat: Remove sections from db configs - https://phabricator.wikimedia.org/T263127 (10Kormat) [13:55:38] 10DBA, 10Security-Team, 10Performance Issue, 10SecTeam Discussion, 10Security: Deploy access to performance_schema/sys for the administrative mediawiki account (mediawiki deployers) - https://phabricator.wikimedia.org/T195578 (10jcrespo) Security never came back with comments, and we gave them a full yea... [13:57:37] 10DBA, 10Security-Team, 10Performance Issue, 10SecTeam Discussion, 10Security: Deploy access to performance_schema/sys for the administrative mediawiki account (mediawiki deployers) - https://phabricator.wikimedia.org/T195578 (10Marostegui) I think it is ok to give access to @Ladsgroup (NDA signed, full... [14:01:05] 10DBA, 10Security-Team, 10Performance Issue, 10SecTeam Discussion, 10Security: Deploy access to performance_schema/sys for the administrative mediawiki account (mediawiki deployers) - https://phabricator.wikimedia.org/T195578 (10jcrespo) @Marostegui Allow me to deploy for now only the access he needs (wi... [14:01:36] 10DBA, 10Security-Team, 10Performance Issue, 10SecTeam Discussion, 10Security: Deploy access to performance_schema/sys for the administrative mediawiki account (mediawiki deployers) - https://phabricator.wikimedia.org/T195578 (10Marostegui) >>! In T195578#6470505, @jcrespo wrote: > @Marostegui Allow me t... [14:39:55] 10DBA, 10Security-Team, 10Performance Issue, 10SecTeam Discussion, 10Security: Deploy access to performance_schema/sys for the administrative mediawiki account (mediawiki deployers) - https://phabricator.wikimedia.org/T195578 (10chasemp) #security-team dropped the ball here on feedback. This is a bit of... [14:45:58] marostegui: [14:46:06] https://www.irccloud.com/pastebin/SyxqHnvA/ [14:46:21] new IP...if you want to make the change now [14:46:41] cmjohnson1: nice! yes [14:47:05] cmjohnson1: I will change it and power the host off. can you do the DNS change yourself? [14:47:35] okay, yes I can but let me know once you power off so I can run script [14:47:51] yep, wildo, doing the ip change now, shouldn't take long [14:49:56] cmjohnson1: I don't see that IP in netbox, was it a dry run? [14:51:12] volans I see ti [14:51:13] it [14:51:39] I see it now, but was definitely not there when I looked (I just refreshed) [14:51:44] cmjohnson1: host off [14:51:52] I *really* hope it's not netbox redis caching [14:56:01] marostegui do you want this 1G or 10G? [14:56:39] cmjohnson1: 1G is good [15:09:55] 10DBA, 10Data-Services: Prepare and check storage layer for arbcom_ruwiki - https://phabricator.wikimedia.org/T262832 (10Urbanecm) Thanks Manuel! [15:56:02] marostegui moved and I am able to ssh in [15:56:16] cmjohnson1: I still see the old dns, was that changed already? [15:57:02] yes [15:57:03] I am on the host via idrac already, so it is nicely up with the old address, but the dns didn't yet spread I guess? [15:57:21] If i can ssh then the new IP is working [15:57:27] cmjohnson1: ok, then let's wait for it to be spread :) [15:57:27] thanks [15:57:28] completely different vlan [15:57:48] yeah, it is spreading now [15:57:55] I can see it changing sometimes, so just a matter of time [15:58:00] I will take it from here, thank you cmjohnson1! [15:59:45] YW! [18:20:10] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: dbproxy1020 PS disconnected - https://phabricator.wikimedia.org/T262998 (10RobH) So this shows an alarm in software, but Chris checked and is green on the actual PSU. I sent a racreset, and it seems to have fixed things: > 23 $> ssh root@dbproxy1020.mgmt.eqia... [18:20:25] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: Tue, Sept 15 PDU Upgrade 12pm-4pm UTC- Racks C4 and C5 - https://phabricator.wikimedia.org/T261456 (10RobH) [18:20:30] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: dbproxy1020 PS disconnected - https://phabricator.wikimedia.org/T262998 (10RobH) 05Open→03Resolved [18:24:01] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: dbproxy1020 PS disconnected - https://phabricator.wikimedia.org/T262998 (10wiki_willy) Thanks @RobH for looking and fixing this. Thanks, Willy [18:29:29] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: dbproxy1020 PS disconnected - https://phabricator.wikimedia.org/T262998 (10Marostegui) Thank you guys!