[01:26:36] <wikibugs>	 10DBA, 10SDC Engineering, 10SDC General, 10Wikidata, and 2 others: Create a production test wiki in group0 to parallel Wikimedia Commons - https://phabricator.wikimedia.org/T197616 (10MusikAnimal) Is this intentionally missing from the replicas?  ` MariaDB [testwiki_p]> USE testcommonswiki_p; ERROR 1044 (4...
[02:39:50] <wikibugs>	 10DBA, 10SDC Engineering, 10SDC General, 10Wikidata, and 2 others: Create a production test wiki in group0 to parallel Wikimedia Commons - https://phabricator.wikimedia.org/T197616 (10Reedy) >>! In T197616#4876205, @MusikAnimal wrote: > Is this intentionally missing from the replicas? >  > ` > MariaDB [tes...
[02:49:27] <wikibugs>	 10DBA, 10SDC Engineering, 10SDC General, 10Wikidata, and 2 others: Create a production test wiki in group0 to parallel Wikimedia Commons - https://phabricator.wikimedia.org/T197616 (10Krenair) >>! In T197616#4876222, @Reedy wrote: >>>! In T197616#4876205, @MusikAnimal wrote: >> Is this intentionally missin...
[03:00:22] <wikibugs>	 10DBA, 10SDC Engineering, 10SDC General, 10Wikidata, and 2 others: Create a production test wiki in group0 to parallel Wikimedia Commons - https://phabricator.wikimedia.org/T197616 (10MusikAnimal) > Don’t think anyone has filed a task requesting the views on labs to be created. It doesn’t happen automatica...
[07:29:32] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['pc1007.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-a...
[08:29:55] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['pc1007.eqiad.wmnet'] `  and were **ALL** successful.
[08:33:29] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10Marostegui) pc1007 got installed and looks good: ` root@pc1007:~# megacli -LDPDInfo -aAll  Adapter #0  Number of Virtual Disks: 1 Virtual Drive: 0 (Target Id: 0) Name...
[08:33:49] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10Marostegui)
[08:33:57] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10Marostegui) 05Open→03Resolved
[09:06:00] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: db1115 (tendril DB) had OOM for some processes and some hw (memory) issues - https://phabricator.wikimedia.org/T196726 (10Marostegui) @Cmjohnson can we request a new DIMM to Dell?
[10:18:54] <wikibugs>	 10DBA, 10Operations: correctable memory errors db1068 (commons primary master database) - https://phabricator.wikimedia.org/T213664 (10jcrespo) I created to track it, it has gone up to 21 since yesterday. We have to consider the possibility of it crashing due to uncorrectable errors and be prepared for a failo...
[10:27:44] <jynus>	 do you need help with dbstore1002 ?
[10:28:00] <marostegui>	 No, so far I am finishing the aria_chk
[10:28:12] <marostegui>	 We'll see what we get
[10:28:24] <marostegui>	 if that doesn't work, I will try to start without aria, and then convert the existing aria tables to innodb
[10:28:28] <marostegui>	 I want to avoid restarting the host
[10:28:33] <marostegui>	 (although I think it is a FS issue)
[10:28:43] <jynus>	 ok, standing by because based on your helpful debugging I would know what to do- I suffered a very similar issue some time ago
[10:28:57] <jynus>	 (Aria crashing)
[10:28:57] <marostegui>	 oh
[10:29:04] <marostegui>	 advise! :)
[10:29:31] <jynus>	 well, the nuclear option- removing aria, but you were also going to do that
[10:29:35] <marostegui>	 yeah
[10:29:36] <marostegui>	 XD
[10:29:39] <marostegui>	 I wanted to leave it to the end
[10:29:40] <jynus>	 removing the logs, let it crash
[10:29:49] <jynus>	 I couldn't do anything else
[10:30:08] <jynus>	 of course, you can copy away the files, the structure is very similar to that of myisam
[10:30:13] <marostegui>	 I will finish the aria check, then remove the aria logs, (as suggested by the error log itself) and see what we get
[10:30:19] <jynus>	 so they can recover on an individual basis
[10:30:23] <jynus>	 *be recovered
[10:30:43] <marostegui>	 we don't have may myisam tables on the wikis, but the staging db does
[10:31:00] <marostegui>	 So I want to avoid starting without aria if possible as that means altering all those tables 
[10:33:06] <elukey>	 (people using dbstore1002 should already have an email about the issue + task)
[10:33:28] <jynus>	 marostegui: oh, I was suggesting to do it by moving out the tables
[10:33:43] <jynus>	 so the server is up and we can then recover those in an individual bases
[10:33:51] <marostegui>	 aaah 
[10:33:52] <jynus>	 for example, the linter tables can be reimported
[10:33:54] <marostegui>	 I didn't get that :)
[10:33:56] <marostegui>	 yeah
[10:34:05] <marostegui>	 but not the staging ones :(
[10:34:32] <jynus>	 I will let you handle it, just saying another dbstore crashed before so if you need help, I may be able to
[10:34:39] <marostegui>	 Thanks! :)
[10:34:41] <marostegui>	 Appreciate it!
[10:34:51] <marostegui>	 btw, will you clean up the test db from sanitarium or should I?
[10:47:27] <wikibugs>	 10DBA, 10SDC Engineering, 10SDC General, 10Wikidata, and 2 others: Create a production test wiki in group0 to parallel Wikimedia Commons - https://phabricator.wikimedia.org/T197616 (10jcrespo) First of all, I am only commenting because I have more information, but access handling is owned by the cloud team...
[11:02:25] <marostegui>	 jynus: meeting time?
[14:10:40] <wikibugs>	 10DBA: Purchase and setup remaining hosts for database backups - https://phabricator.wikimedia.org/T213406 (10jcrespo)
[14:39:23] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review: correctable memory errors db1068 (commons primary master database) - https://phabricator.wikimedia.org/T213664 (10CDanis) p:05Triage→03High
[14:52:50] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review: Implement parsercache service on pc[12]0(07|08|09|10) and replace leased pc[12]00[456] - https://phabricator.wikimedia.org/T208383 (10Marostegui) pc1007 is now up and replicating. It is catching up. Tomorrow I will replace pc1010 with pc1007 for consistency with codfw...
[15:01:23] <wikibugs>	 10DBA, 10Patch-For-Review: Drop tag_summary table - https://phabricator.wikimedia.org/T212255 (10Marostegui)
[15:36:05] <jynus>	 what is the plan with pc hosts, 2 pc1 replicating one from other?
[15:36:44] <marostegui>	 yeah the spare host is replicating from pc1, just for the sake of replicate from somewhere
[15:37:03] <jynus>	 yeah, no issue, just I didn't know what was the plan
[15:37:07] <marostegui>	 yeah
[15:37:11] <marostegui>	 I am going to promote pc1007 to master
[15:37:14] <marostegui>	 to match codfw config
[15:37:17] <jynus>	 I guess any host would work
[15:37:25] <marostegui>	 1007 - pc1
[15:37:27] <marostegui>	 1008 pc2
[15:37:30] <marostegui>	 1009 pc3
[15:37:32] <jynus>	 any shard I mean
[15:37:32] <marostegui>	 1010 pc1
[15:37:41] <jynus>	 yeah, cool to me
[15:37:54] <marostegui>	 right now on eqiad pc1, pc1010 is the master as that one was installed before
[15:37:58] <jynus>	 and pc2007 the same and replicating from 1007 ?
[15:38:03] <marostegui>	 so I will promote pc1007 tomorrow so both are the same
[15:38:27] <jynus>	 another question to know what is pending
[15:38:30] <marostegui>	 pc2007 replicates from pc1010 for now, and tomorrow will be done
[15:38:47] <jynus>	 did you change mw config/cron to increase usage or no change yet?
[15:38:55] <marostegui>	 the retention period you mean?
[15:38:58] <jynus>	 yes
[15:38:59] <marostegui>	 like the TTL
[15:39:12] <marostegui>	 no, I was waiting for the holidays season to be done and now to get pc1007 in place
[15:39:19] <jynus>	 which was more reasonable
[15:39:20] <marostegui>	 so I will push the TTL this week most likely
[15:39:38] <jynus>	 I just want to track what is missing so we do't forget
[15:39:44] <jynus>	 it can wait
[15:39:49] <marostegui>	 yeah, it is all tracked on the tasks
[15:40:07] <marostegui>	 https://phabricator.wikimedia.org/T208383#4877301
[15:40:15] <jynus>	 but I would like to find a good balance between resource usage and room for accidents
[15:40:28] <marostegui>	 https://phabricator.wikimedia.org/T210992#4873982
[15:40:38] <jynus>	 ah, cool
[15:40:43] <jynus>	 so there was already a ticket
[15:40:47] <marostegui>	 yep
[15:40:49] <marostegui>	 and after that
[15:40:52] <marostegui>	 we can start with
[15:41:03] <marostegui>	 https://phabricator.wikimedia.org/T210725
[15:41:07] <jynus>	 please involve me
[15:41:14] <jynus>	 if you want to do it yourself
[15:41:22] <marostegui>	 I will involve you for sure
[15:41:25] <marostegui>	 it is a delicate operation
[15:41:29] <jynus>	 I want to give a deep look at the past and current state
[15:41:52] <marostegui>	 I will actively ask for a +1 for those changes, TTL and key changes anyways
[15:42:01] <jynus>	 I am not sure if to take over T210725 now
[15:42:01] <stashbot>	 T210725: Replace parsercache keys to something more meaningful on db-XXXX.php - https://phabricator.wikimedia.org/T210725
[15:42:14] <jynus>	 on one way, it is better to do it now
[15:42:24] <marostegui>	 before increasing ttl?
[15:42:30] <jynus>	 on the other, we may be mixing 2 changes
[15:42:41] <marostegui>	 yeah, I think we should isolate those
[15:42:46] <marostegui>	 whichever we do first, I don't mind
[15:42:56] <marostegui>	 I think TTL should be first though, as we "have been there"
[15:42:57] <jynus>	 let's send to SOS some request for help
[15:43:17] <jynus>	 to performance and core
[15:43:23] <marostegui>	 for which change?
[15:43:38] <jynus>	 both in general "upcoming parser cache changes"
[15:43:44] <marostegui>	 they are involved on both
[15:43:50] <marostegui>	 I guess we need to decide which one to go first
[15:43:51] <jynus>	 also service ops will want to get involved
[15:44:08] <marostegui>	 I think we should do the TTL first
[15:44:12] <jynus>	 so not sure if SOS, but let's highlight it
[15:44:12] <marostegui>	 As it is a situation we know
[15:44:17] <jynus>	 something is happening soon
[15:44:39] <jynus>	 "be aware of this change(s)"
[15:45:19] <mark>	 hey, do you guys want to do a meeting about the DBA JD tomorrow?
[15:45:32] <marostegui>	 added to the SRE meeting
[15:45:37] <marostegui>	 mark: fine by me
[15:45:37] <jynus>	 ok to me, marostegui
[15:45:39] <jynus>	 *mark
[15:45:43] <mark>	 tomorrow afternoon?
[15:45:46] <jynus>	 ok
[15:45:50] <marostegui>	 sounds good
[15:45:56] <mark>	 from what time are you available after lunch?
[15:46:14] <jynus>	 for me normally 3 is ok
[15:46:17] <marostegui>	 I am free after 3pm all the time
[15:46:18] <jynus>	 *16 CET
[15:46:21] <jynus>	 *15
[15:46:25] <mark>	 let's do 3 then
[15:46:26] <jynus>	 so 14 UTC
[15:46:32] <marostegui>	 mark: ok for me
[15:47:50] <jynus>	 we discussed already some thing today manuel an me
[15:47:55] <jynus>	 *things
[15:59:01] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Marostegui)
[16:00:55] <jynus>	 marostegui: do you have time for backups now or do I go for the s4 restart?
[16:01:36] <marostegui>	 I have time
[16:01:42] <marostegui>	 we can also do it tomorrow morning
[16:01:45] <marostegui>	 whatever you prefer
[16:03:11] <jynus>	 it is just 5 minutes
[16:03:20] <marostegui>	 famous last words!
[16:03:25] <marostegui>	 let's do it then
[16:03:47] <jynus>	 it is 5 minutes to explain, the discussion it is out of my hand
[16:03:51] <marostegui>	 haha
[17:00:13] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: db1115 (tendril DB) had OOM for some processes and some hw (memory) issues - https://phabricator.wikimedia.org/T196726 (10Cmjohnson) @Marostegui I have to do move the DIMM to another slot and see if the error corrects itself moves with the DIMM or remains the same.  Can you...
[17:00:55] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: db1115 (tendril DB) had OOM for some processes and some hw (memory) issues - https://phabricator.wikimedia.org/T196726 (10Marostegui) Yep, we can do that! Just ping us when you are ready for it Thanks!
[17:02:03] <wikibugs>	 10DBA, 10User-Banyek: BBU problems dbstore2002 - https://phabricator.wikimedia.org/T205257 (10Marostegui) a:05Papaul→03None Nothing for Papaul to do here for now.
[17:46:51] <marostegui>	 looks like we need to prepare an s3 failover! 
[19:13:45] <wikibugs>	 10DBA, 10Analytics, 10Operations, 10ops-eqiad: swap a2-eqiad PDU with on-site spare - https://phabricator.wikimedia.org/T213748 (10RobH)
[19:15:09] <wikibugs>	 10DBA, 10Analytics, 10Operations, 10ops-eqiad: swap a2-eqiad PDU with on-site spare - https://phabricator.wikimedia.org/T213748 (10RobH)
[19:18:37] <wikibugs>	 10DBA, 10Analytics, 10Operations, 10ops-eqiad: swap a2-eqiad PDU with on-site spare - https://phabricator.wikimedia.org/T213748 (10RobH)
[19:35:23] <wikibugs>	 10DBA, 10Analytics, 10Operations, 10ops-eqiad: swap a2-eqiad PDU with on-site spare - https://phabricator.wikimedia.org/T213748 (10RobH)
[19:35:55] <wikibugs>	 10DBA, 10Analytics, 10Operations, 10ops-eqiad: swap a2-eqiad PDU with on-site spare - https://phabricator.wikimedia.org/T213748 (10RobH)
[19:51:41] <wikibugs>	 10DBA, 10Analytics, 10Operations, 10ops-eqiad: swap a2-eqiad PDU with on-site spare - https://phabricator.wikimedia.org/T213748 (10RobH) Please note the work has now been scheduled for Thursday, 2019-01-17 @ 07:00 EST (12:00 GMT).  As both the #dba team and the #analytics team have expressed interest in st...
[19:58:03] <wikibugs>	 10DBA, 10Analytics, 10Operations, 10ops-eqiad: swap a2-eqiad PDU with on-site spare - https://phabricator.wikimedia.org/T213748 (10RobH)
[19:58:45] <wikibugs>	 10DBA, 10Analytics, 10Operations, 10ops-eqiad: swap a2-eqiad PDU with on-site spare - https://phabricator.wikimedia.org/T213748 (10Marostegui) @robh what's your plan with db1075 (the db master)?
[19:59:58] <wikibugs>	 10DBA, 10Analytics, 10Operations, 10ops-eqiad: swap a2-eqiad PDU with on-site spare - https://phabricator.wikimedia.org/T213748 (10RobH) >>! In T213748#4878612, @Marostegui wrote: > @robh what's your plan with db1075 (the db master)?   @cmjohnson will take 1 of the 2 power supplies and cross-cable it into...
[20:04:19] <wikibugs>	 10DBA, 10Analytics, 10Operations, 10ops-eqiad: swap a2-eqiad PDU with on-site spare - https://phabricator.wikimedia.org/T213748 (10Marostegui) Awesome! Thanks for clarifying!
[22:57:33] <wikibugs>	 10DBA, 10SDC Engineering, 10SDC General, 10Wikidata, and 2 others: Create a production test wiki in group0 to parallel Wikimedia Commons - https://phabricator.wikimedia.org/T197616 (10Jdforrester-WMF) >>! In T197616#4876226, @MusikAnimal wrote: >> Don’t think anyone has filed a task requesting the views on...