[09:34:38] db2053 had its ilo failing remote ipmi, I reseted through ssh and it came back [09:44:36] 10DBA, 10Patch-For-Review: Test MySQL 8.0 with production data and evaluate its fit for WMF databases - https://phabricator.wikimedia.org/T193226 (10jcrespo) [09:44:42] 10DBA, 10MediaWiki-Database, 10Operations, 10Patch-For-Review: Evaluate and decide the future of relational datastore at WMF after the upgrade of MariaDB 10.1 is finished - https://phabricator.wikimedia.org/T193224 (10jcrespo) [09:44:58] 10DBA, 10Patch-For-Review: Test MySQL 8.0 with production data and evaluate its fit for WMF databases - https://phabricator.wikimedia.org/T193226 (10jcrespo) [11:18:35] 10DBA: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10Zoranzoki21) >>! In T212346#4845775, @Zoranzoki21 wrote: >>>! In T212346#4845733, @MarcoAurelio wrote: >> @Marostegui Given that db67de7fad isn't yet live in WMF branches, perhaps we could wait until it is there so w... [11:39:25] es2017 required a manual powercycle to reach a successful reboot FYI [12:22:09] 10DBA: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10MarcoAurelio) I can give this another try. [12:23:10] 10DBA: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10Zoranzoki21) >>! In T212346#4872682, @MarcoAurelio wrote: > I can give this another try. I will tell you when you can try. I will start now. [12:29:55] 10DBA: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10MarcoAurelio) Okay. [12:40:55] 10DBA: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10Zoranzoki21) >>! In T212346#4872695, @MarcoAurelio wrote: > Okay. @MarcoAurelio I tryed, but I still have problems. As generator for pages which should be deleted use contributions from user Millbot-Weather. As summ... [12:44:31] 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10User-Ladsgroup: Drop change_tag.ct_tag column in production - https://phabricator.wikimedia.org/T210713 (10Ladsgroup) It's cut to half: ` wikiadmin@10.64.32.114(ruwiki)> SELECT table_schema, table_name AS `Table`, round(((data_length + index_length) /... [12:48:04] 10DBA: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10MarcoAurelio) I see that you're being able to delete some of those pages. Please note that pages with more than one thousand of revisions are not deleted instantly but via a JobQueue job so it can take some time unti... [12:50:00] 10DBA: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10Zoranzoki21) >>! In T212346#4872712, @MarcoAurelio wrote: > I see that you're being able to delete some of those pages. Please note that pages with more than one thousand of revisions are not deleted instantly but vi... [12:50:56] 10DBA: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10Zoranzoki21) >>! In T212346#4872712, @MarcoAurelio wrote: > Please note that pages with more than one thousand of revisions are not deleted instantly but via a JobQueue job so it can take some time until the deletion... [13:32:22] 10DBA: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10Zoranzoki21) @MarcoAurelio I generated you new list for first part of deletion. See https://sr.wikinews.org/wiki/Корисник:MarcoAurelio/ToDel You can start when you feel conform to do it. [13:34:35] 10DBA, 10Serbian-Sites: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10Liuxinyu970226) [13:58:49] 10DBA, 10Serbian-Sites: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10MarcoAurelio) Thank you. I'll try to delete those pages then. To avoid flooding recent changes, do you think I can get a local bot flag at sr.wikinews? Ping @dungodung as one of the local bureaucrats. [14:03:05] 10DBA, 10Serbian-Sites: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10Zoranzoki21) >>! In T212346#4872809, @MarcoAurelio wrote: > Thank you. I'll try to delete those pages then. To avoid flooding recent changes, do you think I can get a local bot flag at sr.wikinews?... [15:20:45] 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10User-Ladsgroup: Drop change_tag.ct_tag column in production - https://phabricator.wikimedia.org/T210713 (10Marostegui) Keep in mind that this also triggered a table rebuilt, so the table got defragmented too [15:32:05] marostegui: something strange happened on db1115: https://grafana.wikimedia.org/d/000000274/prometheus-machine-stats?orgId=1&var-server=db1115&var-datasource=eqiad%20prometheus%2Fops&from=1539188224828&to=1547220604061 [15:32:33] total memory available went from 128 to 96 GB [15:32:43] "total memory" should never change [15:39:00] 10DBA: db1115 (tendril DB) had OOM for some processes - https://phabricator.wikimedia.org/T196726 (10jcrespo) 05Resolved→03Open a:05Marostegui→03None Something else happened on the 17th: {F27855028} Total memory went down to 96GB: ` root@db1115:~$ free -m total used free... [15:50:11] 10DBA, 10Operations, 10ops-eqiad: db1115 (tendril DB) had OOM for some processes - https://phabricator.wikimedia.org/T196726 (10jcrespo) a:03Cmjohnson Asking @Cmjohnson to move around this memory module- either it got disconnected or broken completely- FYI 128 GB of memory should be detected, but only 96 a... [15:54:32] 10DBA, 10Operations, 10ops-eqiad: db1115 (tendril DB) had OOM for some processes and some hw (memory) issues - https://phabricator.wikimedia.org/T196726 (10jcrespo) [15:57:08] 10DBA, 10Operations, 10ops-eqiad: db1115 (tendril DB) had OOM for some processes and some hw (memory) issues - https://phabricator.wikimedia.org/T196726 (10jcrespo) More logs, confirming the module is probably dead: `lines=10 2018-12-16T07:40:35-0600 PR8 **Device not detected: DDR4 DIMM(Socket B2)**... [16:26:35] 10DBA, 10Operations, 10ops-eqiad: db1115 (tendril DB) had OOM for some processes and some hw (memory) issues - https://phabricator.wikimedia.org/T196726 (10Marostegui) This host should also be under warranty I guess? [16:30:35] jynus: good catch, I missed that fact [16:38:14] 10DBA, 10Operations, 10ops-eqiad: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10Marostegui) @RobH I can do the OS installation once you give me the green light for it. [16:55:58] 10DBA, 10Serbian-Sites: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10dungodung) I don't see a need for deleting those pages. [17:02:26] 10DBA, 10Serbian-Sites: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10MarcoAurelio) >>! In T212346#4873528, @dungodung wrote: > I don't see a need for deleting those pages. Thanks for your comment @dungodung - On meta they said there was a community discussion. Neve... [17:04:42] 10DBA, 10Serbian-Sites: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10dungodung) As you can see on [[sr:n:Викивести:Форум]] there was no discussion about this at all. Also, I wouldn't consider a few disparate users that contribute here and there a community... [17:17:23] 10DBA, 10Serbian-Sites: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10Zoranzoki21) >>! In T212346#4873537, @dungodung wrote: > As you can see on [[sr:n:Викивести:Форум]] there was no discussion about this at all. Also, I wouldn't consider a few disparate users that c... [17:18:29] 10DBA, 10Serbian-Sites: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10dungodung) Please don't troll here. [17:19:47] 10DBA, 10Serbian-Sites: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10Zoranzoki21) >>! In T212346#4873549, @dungodung wrote: > Please don't troll here. Again: > When you make mistakes, noone must not tell you something bad to you. When we make mistakes, you are fi... [17:24:12] 10DBA, 10Serbian-Sites: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10Zoranzoki21) 05Open→03Stalled I will start public discussion so.. [17:28:56] 10DBA, 10Serbian-Sites: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10jcrespo) @Zoranzoki21 and @dungodung, as well as other subscribers- Phabricator is not the place for this kind of discussion- that should go to wiki. As @MarcoAurelio said, no deletion will happen... [17:35:42] 10DBA, 10Serbian-Sites: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10Zoranzoki21) >>! In T212346#4873594, @jcrespo wrote: > @Zoranzoki21 and @dungodung, as well as other subscribers- Phabricator is not the place for this kind of discussion- that should go to wiki. A... [18:32:09] 10DBA, 10Serbian-Sites: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10Aklapper) @dungodung, @Zoranzoki21: This tone is not welcome here. Please read https://www.mediawiki.org/wiki/Bug_management/Phabricator_etiquette if you would like to be active on Wikimedia Phabri... [18:36:43] 10DBA, 10Serbian-Sites: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10Zoranzoki21) >>! In T212346#4873760, @Aklapper wrote: > @dungodung, @Zoranzoki21: This tone is not welcome here. Please read https://www.mediawiki.org/wiki/Bug_management/Phabricator_etiquette if y... [18:48:00] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10Marostegui) [19:19:16] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10Ladsgroup) [19:27:14] 10DBA, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Marostegui) @elukey as per our chat past week I would like to get at least one host up and ready and with data in plac... [19:35:22] 10DBA, 10Data-Services, 10Datasets-General-or-Unknown, 10User-notice: Archive and drop education program (ep_*) tables on all wikis - https://phabricator.wikimedia.org/T174802 (10Marostegui) Hey! So as per https://meta.wikimedia.org/wiki/Tech/News/2018/47 are we good to go now and drop these tables? Realis... [20:37:18] 10DBA, 10Cloud-Services, 10cloud-services-team, 10User-Banyek: Prepare and check storage layer for nap.wikisource - https://phabricator.wikimedia.org/T210762 (10Marostegui) a:05Banyek→03None Once this is done, let us (DBAs) know and will sanitize it Thanks! [20:37:36] 10DBA, 10Patch-For-Review, 10User-Banyek: Productionize dbproxy101[2-7].eqiad.wmnet - https://phabricator.wikimedia.org/T202367 (10Marostegui) a:05Banyek→03None [20:37:55] 10DBA, 10Patch-For-Review, 10User-Banyek, 10Wikimedia-Incident: Compare a few tables per section between hosts and DC - https://phabricator.wikimedia.org/T207253 (10Marostegui) a:05Banyek→03None [20:39:37] 10DBA, 10Operations, 10Performance-Team: Increase parsercache keys TTL from 22 days back to 30 days - https://phabricator.wikimedia.org/T210992 (10Marostegui) So we are close to get the last host ready {T207258}, once that is done I will revert everything back to 30 days. [20:41:57] 10DBA, 10Operations, 10Patch-For-Review, 10User-Banyek: Implement parsercache service on pc[12]0(07|08|09|10) and replace leased pc[12]00[456] - https://phabricator.wikimedia.org/T208383 (10Marostegui) We are waiting on Robh to finish the last DCOps steps so we can get these installed and ready by, hopeful... [20:42:40] 10DBA, 10Operations, 10Patch-For-Review: Implement parsercache service on pc[12]0(07|08|09|10) and replace leased pc[12]00[456] - https://phabricator.wikimedia.org/T208383 (10Marostegui) 05Stalled→03Open [20:48:05] 10DBA, 10Operations, 10ops-eqiad: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10RobH) production dns updated NIC.Embedded.1-1-1 Ethernet = D0:94:66:75:D1:63 [20:55:37] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10Marostegui) [21:39:54] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10RobH) a:05RobH→03Marostegui [23:26:58] 10DBA, 10Data-Services, 10Datasets-General-or-Unknown, 10User-notice: Archive and drop education program (ep_*) tables on all wikis - https://phabricator.wikimedia.org/T174802 (10greg) >>! In T174802#4873874, @Marostegui wrote: > Hey! > So as per https://meta.wikimedia.org/wiki/Tech/News/2018/47 are we goo...