[05:25:27] <wikibugs_>	 10DBA: Productionize 8 eqiad hosts - https://phabricator.wikimedia.org/T192979#4156169 (10Marostegui) p:05Triage>03Normal
[05:26:15] <wikibugs_>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack and setup db1116 - db1123 - https://phabricator.wikimedia.org/T191792#4156182 (10Marostegui)
[05:27:45] <wikibugs_>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack and setup db1116 - db1123 - https://phabricator.wikimedia.org/T191792#4116638 (10Marostegui)
[05:35:14] <wikibugs_>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack and setup db1116 - db1123 - https://phabricator.wikimedia.org/T191792#4156186 (10Marostegui)
[05:36:02] <wikibugs_>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack and setup db1116 - db1123 - https://phabricator.wikimedia.org/T191792#4116638 (10Marostegui) @Cmjohnson I have confirmed that all the hosts with the exception of db1120 as you mentioned, are up and ready - let's keep this opened till it is fixed. T...
[06:35:40] <jynus>	 ERROR 1799 (HY000): Creating index 'PRIMARY' required more than 'innodb_online_alter_log_max_size' bytes of modification log. Please try again.
[06:35:57] <marostegui>	 Ah yeah, that is a known issue to me too
[07:34:42] <jynus>	 there is prometheus running for all sections on db1115
[07:34:51] <jynus>	 *db1116
[07:35:01] <jynus>	 should I kill a few?
[07:35:19] <marostegui>	 Oh, I will take care of it
[07:35:23] <marostegui>	 thanks for letting me know!
[07:35:50] <jynus>	 when removing section, you need to delete files on several locations
[07:36:08] <jynus>	 it is not done automatically
[07:37:15] <marostegui>	 I guess the prometheus exporters?
[07:37:49] <jynus>	 also for mysql
[07:37:59] <marostegui>	 yeah, I removed those already
[07:38:03] <jynus>	 mostly config on etc
[07:39:04] <jynus>	  /etc/default/prometheus-mysqld-exporter*
[07:39:13] <marostegui>	 yeah, just did those :)
[07:40:07] <jynus>	 I mention it because systemctl was alerting
[07:40:13] <marostegui>	 yeah yeah
[07:40:16] <marostegui>	 I am glad you did
[07:40:45] <jynus>	 I can add the ensure => absent
[07:40:55] <jynus>	 but I didn't want in case of accidental deletion
[07:41:48] <jynus>	 I can setup other host in core if you are ok with that
[07:41:51] <marostegui>	 yeah, I think it is better not to
[07:41:54] <marostegui>	 jynus: sure!
[07:41:57] <marostegui>	 Let me update the new ticket
[07:42:33] <wikibugs_>	 10DBA, 10Patch-For-Review: Productionize 8 eqiad hosts - https://phabricator.wikimedia.org/T192979#4156316 (10Marostegui)
[07:45:07] <wikibugs_>	 10DBA, 10Patch-For-Review: Productionize 8 eqiad hosts - https://phabricator.wikimedia.org/T192979#4156319 (10jcrespo)
[08:15:29] <wikibugs_>	 10DBA, 10Patch-For-Review: Productionize 8 eqiad hosts - https://phabricator.wikimedia.org/T192979#4156401 (10jcrespo)
[08:21:49] <wikibugs_>	 10DBA, 10Operations-Software-Development: Debmonitor: request for misc DB allocation - https://phabricator.wikimedia.org/T192875#4156409 (10Marostegui) I am fine placing this on m2 as Jaime originally suggested.
[08:22:55] <jynus>	 I am chosing s2 because it has 2 "old" servers, and db1122 because it has D row,of which there is none at s2
[08:23:16] <marostegui>	 So we will be able to free up the vslow host there and move it to misc?
[08:23:46] <marostegui>	 Maybe you can also use it for s4, which has the other old host
[08:24:01] <jynus>	 not sure if I will place it as multiinstance
[08:24:26] <marostegui>	 Why not?
[08:26:18] <jynus>	 there is too many D hosts on s4
[08:26:51] <marostegui>	 maybe for s7 then?
[08:27:01] <jynus>	 I prefer to convert an existing s2 host from C
[08:27:09] <marostegui>	 ah sure, that works too
[08:27:55] <jynus>	 it is a triple move, but I think it will be a nicer setup
[08:28:17] <marostegui>	 yeah
[08:28:19] <jynus>	 22 substitutes 90 which substitutes db1060
[08:28:28] <marostegui>	 haha
[08:28:53] <marostegui>	 yeah, makes sense
[08:28:54] <jynus>	 I am not sure about that either, maybe s4 needs dedicated vslows
[08:29:10] <marostegui>	 right now it is a "slow" host
[08:29:13] <jynus>	 So I am going to pool it, remove old hosts
[08:29:23] <marostegui>	 So probably the SSDs will compensate the smaller buffer pool
[08:29:25] <jynus>	 and we can rearrange later
[08:29:33] <marostegui>	 sure
[08:32:08] <jynus>	 leaving it replicating for some days, etc.
[08:32:39] <marostegui>	 yeah
[08:33:19] <jynus>	 and with the dump replicas arriving later, I am not sure multi-instance is the way, at least for all sections
[08:34:00] <jynus>	 and thinking too much ahead... I just want to remove the old servers for now
[08:34:07] <marostegui>	 sure, let's stick to that
[08:34:37] <jynus>	 and fixing the row gaps
[08:39:26] <wikibugs_>	 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#4156461 (10Marostegui)
[09:01:32] <wikibugs_>	 10DBA, 10Dumps-Generation: Some dump hosts are accessing main traffic servers - https://phabricator.wikimedia.org/T143870#4156551 (10jcrespo) `dumpsgen  4973 /usr/bin/php5 /srv/mediawiki/multiversion/MWScript.php dumpBackup.php --wiki=zhwiki` is accessing a main traffic (no `dump`) server at snapshot1007, and...
[09:32:07] <wikibugs_>	 10DBA, 10Operations-Software-Development: Debmonitor: request for misc DB allocation - https://phabricator.wikimedia.org/T192875#4156668 (10jcrespo) @Volans you can speed up the process by setting some password on the private repository (and some non-private equivalent in the labs/private one), and suggesting...
[09:56:08] <jynus>	 should we convert the enwiki remaining myisam table to innodb?
[09:56:22] <marostegui>	 there are still myisam tables there?
[09:56:44] <jynus>	 1
[09:56:50] <marostegui>	 which one is that?
[09:56:55] <jynus>	 bv_editsSOMEYEAR
[09:57:06] <marostegui>	 :|
[09:57:23] <marostegui>	 I guess it is fine to convert them (I wonmder if they are in use)
[09:57:36] <marostegui>	 They are not
[09:57:41] <marostegui>	 per T54921
[09:57:42] <stashbot>	 T54921: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921
[09:57:43] <jynus>	 maybe it is scheduled for deletion?
[09:57:47] <marostegui>	 yep
[09:57:48] <marostegui>	 they are
[09:57:57] <marostegui>	 2009, 2011 and 2015
[09:58:02] <marostegui>	 (don't know if there are more)
[09:58:27] <jynus>	 I think they are
[09:59:53] <marostegui>	 On that task, only 2009 2011 and 2015 appear to be scheduled for deletion
[10:00:20] <jynus>	 when handled, we can create a task with all existing ans ask again just in case
[10:04:54] <marostegui>	 Amir1: the logging table can be optimized already in s8? I was thinking that I have to do an alter table on that table which is currently 430G, and I was wondering if it can be optimized already so we can free up some space there
[10:16:47] <Amir1>	 marostegui: the table is half of its size now
[10:17:02] <Amir1>	 but the plan is to reduce it to 1% of its size
[10:17:23] <Amir1>	 so 300M more rows to delete but 360M already deleted
[10:17:28] <Amir1>	 as you wish :D
[10:17:29] <jynus>	 we have a purging issue that I don't like
[10:17:51] <marostegui>	 Amir1: Then I will wait and proceed with other shards before I hit s8 then :)
[10:17:56] <marostegui>	 When do you think it will be ready for it?
[10:18:45] <jynus>	 we may want to clean up dbstore2001 in advance, marostegui: https://grafana.wikimedia.org/dashboard/db/mysql?orgId=1&var-dc=codfw%20prometheus%2Fops&var-server=dbstore2001&var-port=13318
[10:19:03] <marostegui>	 uf
[10:19:03] <Amir1>	 marostegui: it needs jynus to approve because he said I need to stop deleting 
[10:19:22] <jynus>	 or rebuild it or something
[10:19:26] <jynus>	 or it will explode
[10:19:44] <jynus>	 it may not be the only server
[10:20:04] <jynus>	 I can make it stop, but increasing the purge threads, but then it starts lagging
[10:20:26] <marostegui>	 yeah
[10:20:41] <marostegui>	 it is madness
[10:20:43] <jynus>	 I can try again, now that backups should be complete
[10:20:53] <marostegui>	 They are still running on 2001
[10:21:11] <jynus>	 maybe wikibase has a leak issue
[10:21:13] <marostegui>	 https://grafana.wikimedia.org/dashboard/db/mysql?orgId=1&var-dc=codfw%20prometheus%2Fops&var-server=dbstore2001&var-port=13318&panelId=11&fullscreen&from=now-30d&to=now
[10:21:15] <jynus>	 with too many deletes?
[10:21:17] <marostegui>	 scary
[10:24:35] <marostegui>	 maybe we can just rebuild s8 there
[10:24:59] <jynus>	 let me try rebuilding logging once
[10:25:06] <jynus>	 we have really nothing to lose
[10:25:12] <marostegui>	 rebuilding the table you mean?
[10:25:17] <jynus>	 yes
[10:25:23] <marostegui>	 sounds good
[10:25:24] <jynus>	 and or increasing again the purge threads
[10:25:36] <marostegui>	 I wanted to run an alter on s2 in codfw - should I hold?
[10:25:48] <jynus>	 no problem
[10:25:52] <jynus>	 this is s8
[10:25:59] <jynus>	 if it goes slow, it goes slow
[10:26:04] <marostegui>	 sure, but I didn't want to add more IO there :)
[10:26:44] <marostegui>	 I'll wait for the s2 backups to be done
[10:27:55] <jynus>	 I've started an ALTER TABLE logging ENGINE=InnoDB row_format=Compressed, FORCE;
[10:28:22] <marostegui>	 cool
[10:28:31] <marostegui>	 -rw-rw---- 1 mysql mysql 217G Apr 25 10:20 logging.ibd
[10:28:37] <marostegui>	 let's see how much it shrinks
[10:28:41] <jynus>	 I will stop replication
[10:28:41] <marostegui>	 Amir1: ^
[10:29:55] <Amir1>	 thanks
[10:29:57] <jynus>	 but if it keeps happening it may not be logging but other tables
[10:30:07] <Amir1>	 I'm not deleting anything on s8 for now
[10:30:28] <jynus>	 deletes are very low according to stats, too
[10:30:32] <jynus>	 so it is quite strange
[10:30:47] <jynus>	 that is why I didn't want to add more issues at the moment, Amir1
[10:31:26] <jynus>	 maybe the compression + low io makes it very laggy, still
[10:31:47] <Amir1>	 I trust you fully, if you say stop, there is no need to say more :)
[10:32:17] <jynus>	 Amir1: the thing is, if it continues in that trend, things end up exploding (crashes)
[10:32:44] <Amir1>	 jynus: marostegui I'm deleting logging from dewiki (probably already done) and commons (this is pretty slow but ongoing) do you want me to stop them too?
[10:33:03] <jynus>	 no, those caused no issues that I can see
[10:33:07] <jynus>	 but let me double-check
[10:33:41] <marostegui>	 s4 is fine so far
[10:33:43] <Amir1>	 I have deleted couple of millions of rows on all wikis
[10:34:03] <Amir1>	 all small and medium are already done, half of large wikis as well
[10:34:11] <jynus>	 but it is only s8 with issues
[10:34:18] <jynus>	 all others are ok
[10:34:32] <Amir1>	 noted
[10:35:03] <jynus>	 could be unrelated and be a host only issue, but we need time to solve it
[10:35:16] <jynus>	 will ping you on phabricator when we are back in a good state
[10:36:51] <Amir1>	 is there a phabricator for this issue?
[10:37:29] <jynus>	 no, I will ping you in the "delete logging rows" one
[10:38:36] <jynus>	 marostegui: I got 2 more duplicate errors while compressing dbstore1001
[10:38:44] <jynus>	 ERROR 1062 (23000) at line 1: Duplicate entry '1047393233' for key 'ts_rc_id'
[10:38:50] <jynus>	 ERROR 1062 (23000) at line 1: Duplicate entry 'Q30325899-T-44294723' for key 'eu_entity_id'
[10:39:06] <marostegui>	 can you try those again with replication enabled?
[10:39:16] <jynus>	 doing right now
[10:39:22] <marostegui>	 I will need to compress db1116.s1 so we will see if I get duplicates too
[10:39:36] <jynus>	 tag_summary and wbc_entity_usage
[10:39:53] <jynus>	 both friends of us
[10:40:37] <jynus>	 both with likely ongoing replication activity
[10:43:39] <jynus>	 "sqldata/zhwiki/revision_broken.ibd"
[10:44:06] <marostegui>	 ?
[10:44:24] <jynus>	 lots of garbage on the servers
[10:44:31] <jynus>	 (I guess)
[10:45:11] <marostegui>	 never seen that before
[11:06:30] <jynus>	 the tag_summary error happend again
[11:06:41] <jynus>	 the other hasn't finished yet
[11:06:57] <jynus>	 the key that failed is different, though
[11:07:08] <jynus>	 I think this is an alter table issue
[11:33:12] <jynus>	 https://tendril.wikimedia.org/host/view/db1122.eqiad.wmnet/3306
[11:33:20] <jynus>	 https://grafana.wikimedia.org/dashboard/db/mysql?orgId=1&from=now-1h&to=now&var-dc=eqiad%20prometheus%2Fops&var-server=db1122&var-port=9104
[11:35:32] <jynus>	 I will reimage db1090 after lunch
[12:47:08] <jynus>	 tag summary fails every time, each time on a different id
[12:48:00] <jynus>	 https://phabricator.wikimedia.org/P7038
[12:49:08] <jynus>	 I am going to try once again with replication stopped
[12:49:31] <marostegui>	 pffff
[12:49:41] <marostegui>	 maybe it has too much activity?
[12:50:06] <jynus>	 if it has too much, then it a "normal" error, and it complains about the buffer
[12:50:30] <jynus>	 I think this is a bug
[12:50:39] <marostegui>	 :(
[12:51:04] <jynus>	 in the past I would have said it is old data types or something
[12:51:20] <jynus>	 but this is a newly imported table so to start from 10.1 directly
[12:51:52] <jynus>	 maybe it happens again with replication off
[12:51:56] <marostegui>	 let's see
[12:51:58] <jynus>	 we'll see
[12:53:47] <jynus>	 I think I will move all db1060's roles to db1090
[12:53:52] <jynus>	 the decom db1060
[12:53:55] <jynus>	 *then
[12:54:15] <marostegui>	 cool
[13:00:12] <wikibugs_>	 10DBA, 10Patch-For-Review: Productionize 8 eqiad hosts - https://phabricator.wikimedia.org/T192979#4157419 (10jcrespo)
[13:21:55] <jynus>	 it worked first try with replication stopped
[13:22:28] <marostegui>	 is it worth reporting?
[13:23:07] <jynus>	 I think yes, the problem is to make it reproducible
[13:23:12] <marostegui>	 Yeah, that's the thing
[13:57:35] <wikibugs_>	 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy - https://phabricator.wikimedia.org/T190704#4157635 (10Marostegui)
[14:04:17] <wikibugs_>	 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy - https://phabricator.wikimedia.org/T190704#4157672 (10jcrespo)
[15:29:14] <cmjohnson1>	 @marostegui @jynus db1120 is doing the initial puppet run after that all 8 of the new db's are ready for you.
[15:30:10] <wikibugs_>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack and setup db1116 - db1123 - https://phabricator.wikimedia.org/T191792#4158048 (10Cmjohnson)
[15:30:16] <jynus>	 thank you very much chris!
[15:31:18] <wikibugs_>	 10DBA, 10Operations, 10Patch-For-Review: Rack and setup db1116 - db1123 - https://phabricator.wikimedia.org/T191792#4116638 (10Cmjohnson) a:05Cmjohnson>03Marostegui @marostegui db1120 is fixed, i had the ethernet cable in the wrong port :-(.   Assigning to you and removing ops-eqiad tag
[15:31:24] <cmjohnson1>	 YW...resolve that task if you no longer need it
[15:33:14] <wikibugs_>	 10DBA, 10Operations, 10Patch-For-Review: Rack and setup db1116 - db1123 - https://phabricator.wikimedia.org/T191792#4158081 (10Marostegui) 05Open>03Resolved a:05Marostegui>03Cmjohnson Confirmed db1120 looks good! Thanks @Cmjohnson!
[15:33:28] <wikibugs_>	 10DBA, 10Operations, 10Patch-For-Review: Rack and setup db1116 - db1123 - https://phabricator.wikimedia.org/T191792#4158086 (10Marostegui)
[15:33:49] <wikibugs_>	 10DBA, 10Patch-For-Review: Productionize 8 eqiad hosts - https://phabricator.wikimedia.org/T192979#4158088 (10Marostegui)
[18:33:09] <wikibugs_>	 10DBA, 10Cloud-Services, 10cloud-services-team (Kanban): Prepare storage layer for  euwikisource - https://phabricator.wikimedia.org/T189466#4158969 (10Bstorm) The wiki replica views and DNS are set.
[18:33:42] <wikibugs_>	 10DBA, 10Cloud-Services, 10Hindi-Sites, 10User-Jayprakash12345, 10cloud-services-team (Kanban): Prepare and check storage layer for hiwikimedia - https://phabricator.wikimedia.org/T188490#4158971 (10Bstorm) The wiki replica views and DNS are set. -- and updated that document so it is now correct.
[18:34:04] <wikibugs_>	 10DBA, 10Cloud-Services, 10User-Urbanecm, 10cloud-services-team (Kanban): Prepare and check storage layer for inhwiki - https://phabricator.wikimedia.org/T184375#4158972 (10Bstorm) The wiki replica views and DNS are set.
[18:34:21] <wikibugs_>	 10DBA, 10Cloud-Services, 10User-Urbanecm, 10cloud-services-team (Kanban): Prepare and check storage layer for gorwiki - https://phabricator.wikimedia.org/T189112#4158973 (10Bstorm) The wiki replica views and DNS are set.
[18:34:53] <wikibugs_>	 10DBA, 10Cloud-Services, 10User-Urbanecm, 10cloud-services-team (Kanban): Prepare storage layer for lfnwiki - https://phabricator.wikimedia.org/T183566#4158976 (10Bstorm) The wiki replica views and DNS are set, and corrected mentioned doc.
[18:35:12] <wikibugs_>	 10DBA, 10Cloud-Services, 10User-Urbanecm, 10cloud-services-team (Kanban): Prepare and check storage layer for romdwikimedia - https://phabricator.wikimedia.org/T187774#4158978 (10Bstorm) The wiki replica views and DNS are set.
[18:56:03] <wikibugs_>	 10DBA, 10Cloud-Services, 10cloud-services-team (Kanban): Prepare storage layer for  euwikisource - https://phabricator.wikimedia.org/T189466#4159072 (10Bstorm) 05Open>03Resolved
[18:56:33] <wikibugs_>	 10DBA, 10Cloud-Services, 10Hindi-Sites, 10User-Jayprakash12345, 10cloud-services-team (Kanban): Prepare and check storage layer for hiwikimedia - https://phabricator.wikimedia.org/T188490#4159074 (10Bstorm) 05Open>03Resolved
[18:56:58] <wikibugs_>	 10DBA, 10Cloud-Services, 10User-Urbanecm, 10cloud-services-team (Kanban): Prepare and check storage layer for inhwiki - https://phabricator.wikimedia.org/T184375#4159076 (10Bstorm) 05Open>03Resolved
[18:57:09] <wikibugs_>	 10DBA, 10Cloud-Services, 10User-Urbanecm, 10cloud-services-team (Kanban): Prepare and check storage layer for gorwiki - https://phabricator.wikimedia.org/T189112#4159090 (10Bstorm) 05Open>03Resolved
[18:57:47] <wikibugs_>	 10DBA, 10Cloud-Services, 10User-Urbanecm, 10cloud-services-team (Kanban): Prepare storage layer for lfnwiki - https://phabricator.wikimedia.org/T183566#4159092 (10Bstorm) 05Open>03Resolved
[18:58:09] <wikibugs_>	 10DBA, 10Cloud-Services, 10User-Urbanecm, 10cloud-services-team (Kanban): Prepare and check storage layer for romdwikimedia - https://phabricator.wikimedia.org/T187774#4159095 (10Bstorm) 05Open>03Resolved