[00:30:58] <wikibugs_>	 10DBA, 10Wikidata, 10Performance, 10User-Daniel: DispatchChanges: Avoid long-lasting connections to the master DB - https://phabricator.wikimedia.org/T151681#3389322 (10Ladsgroup)
[00:31:01] <wikibugs_>	 10DBA, 10Wikidata, 10Patch-For-Review, 10Performance, and 3 others: Consider only updating wb_changes_dispatch after a successful run - https://phabricator.wikimedia.org/T162556#3389321 (10Ladsgroup) 05Open>03Resolved
[00:31:04] <wikibugs_>	 10DBA, 10Wikidata, 10Performance, 10User-Daniel: DispatchChanges: Avoid long-lasting connections to the master DB - https://phabricator.wikimedia.org/T151681#2824480 (10Ladsgroup)
[00:31:07] <wikibugs_>	 10DBA, 10Wikidata, 10Patch-For-Review, 10Performance, and 3 others: Use replica for reading the last dispatch position (chd_seen) - https://phabricator.wikimedia.org/T162557#3389323 (10Ladsgroup) 05Open>03Resolved
[00:50:49] <wikibugs_>	 10DBA, 10Wikidata, 10Performance, 10User-Daniel: DispatchChanges: Avoid long-lasting connections to the master DB - https://phabricator.wikimedia.org/T151681#3389346 (10Ladsgroup) With the deployment of the changes in the dispatching, in the last 24 hours we had around 1,800 cases of going readonly but thi...
[05:25:37] <wikibugs_>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s7 - https://phabricator.wikimedia.org/T166208#3389527 (10Marostegui)
[06:56:28] <jynus>	 what work or issue is happening on db1047?
[06:57:31] <jynus>	 I see/know the alter tables, but that would not explain the monitoring problems?
[06:58:13] <marostegui>	 jynus: Only the alter tables - I haven't checked a lot, but maybe overload and the checks timing out?
[06:58:29] <jynus>	 why acking, them?
[06:59:13] <jynus>	 it is ok to leave them off and anyone else can have a look
[06:59:16] <marostegui>	 Because there is work going on and the threads are running finely, so I thought I would downtime them until the work is complete, as they might be flapping
[06:59:24] <jynus>	 it could be the changes I did yesterday
[07:00:31] <jynus>	 it is using the wrong socket
[07:01:28] <marostegui>	 did it get restarted or what?
[07:01:34] <jynus>	 no
[07:01:38] <jynus>	 but for some reason
[07:01:51] <jynus>	 etc says the socket is in one place
[07:01:59] <jynus>	 but icinga searches it on the wrong one
[07:02:17] <jynus>	 however, the alarm didn't happen yesterday
[07:02:28] <jynus>	 at least not hours after I reenabled puppet
[07:04:15] <marostegui>	 Maybe it was already downtimed when that happened and it just cabe back from downtime
[07:04:59] <jynus>	 could be
[07:05:03] <jynus>	 I will try to fix it
[07:05:37] <jynus>	 it is ok to leave things unacked, if you do not know why it is happening/have other things to do
[07:05:50] <jynus>	 we are a team :-D
[07:06:04] <marostegui>	 :)
[07:06:14] <jynus>	 or ack and create a ticket
[07:06:38] <jynus>	 normally I ask you first in case you know
[07:06:48] <jynus>	 like here
[07:06:55] <marostegui>	 Nah, this time I assumed it was the ongoing alters
[07:07:00] <jynus>	 ok ok
[07:07:03] <jynus>	 fair enough
[07:07:16] <marostegui>	 (as I have seen it before, specially with the huge alter table I tried for 13 days (!!) on revision table)
[07:08:25] <jynus>	 I see, it is a very simple mistake
[07:08:34] <jynus>	 just the default socket location has changed
[07:10:21] <jynus>	 I created this
[07:10:49] <jynus>	 but it must be as you said- I didn't realize because downtime or something
[07:11:38] <marostegui>	 yeah, maybe it was downtimed when the alters started
[07:47:20] <jynus>	 x1 is replicating still from codfw
[07:47:24] <jynus>	 but it is on row C
[07:47:50] <jynus>	 es1011 has es2016 as master
[07:48:05] <jynus>	 but on row D
[07:48:18] <marostegui>	 yeah, only row D is supposed to be affected
[07:48:30] <jynus>	 I thought it was A?
[07:48:34] <marostegui>	 These are the hosts I found that might page: https://phabricator.wikimedia.org/T168462#3366437
[07:48:37] <marostegui>	 sorry, A
[07:48:46] <marostegui>	 D was the one done last week
[07:49:26] <jynus>	 es2018 is on row B
[07:49:30] <jynus>	 so also unaffected
[07:49:43] <jynus>	 (I was only chacking replication indirect pages)
[07:50:03] <marostegui>	 yes yes
[07:50:07] <jynus>	 I don't se those being affected
[07:50:28] <jynus>	 s4 is disconnected
[07:50:33] <marostegui>	 no, and even on: https://phabricator.wikimedia.org/T168462#3366437 db1068 isn't affected because replicaiton is disconnected
[07:50:36] <jynus>	 and the other too
[07:50:36] <marostegui>	 yeah
[07:50:44] <jynus>	 plus those don't page
[07:50:55] <jynus>	 misc and codfw dont page for replication
[07:51:06] <jynus>	 because users don't get affected
[07:51:11] <jynus>	 I would down, however
[07:51:39] <jynus>	 all hosts and services for a couple of hours- maybe mysql process and disk space pages
[07:51:59] <jynus>	 what do you think?
[07:52:04] <marostegui>	 why?
[07:52:12] <jynus>	 to avoid paging
[07:52:22] <jynus>	 or at least the one that page
[07:52:43] <marostegui>	 but why mysqld process and disk space would page for this maintenance? (they didn't page for row D one)
[07:52:50] <jynus>	 ok, then
[07:53:07] <marostegui>	 not saying we shouldn't, just trying to understand your thinking process
[07:53:10] <jynus>	 I thought they may do, but it makes sense they will not
[07:53:10] <marostegui>	 :)
[07:53:26] <jynus>	 specially if you tell me they didn't the other time
[07:53:39] <marostegui>	 no, last time only es in eqiad paged
[07:53:46] <jynus>	 I do not trust the service dependency is well configured
[07:53:47] <marostegui>	 for replication broken
[07:53:54] <jynus>	 but from what you say it is
[07:54:10] <jynus>	 I think it is because I am accustumed
[07:54:24] <jynus>	 of when a service goes down, it pagers when it comes back
[07:54:33] <jynus>	 but only because mysql is not working
[07:54:40] <jynus>	 it is a new host
[07:55:38] <marostegui>	 ah yeah
[07:55:38] <jynus>	 there is one extra thing we should check
[07:55:50] <jynus>	 do you know if some es host is affected?
[07:56:15] <marostegui>	 there are no es masters on rown A
[07:56:17] <marostegui>	 row
[07:56:19] <jynus>	 slaves
[07:56:46] <marostegui>	 yes
[07:56:47] <marostegui>	 there were some
[07:56:49] <marostegui>	 let me check
[07:56:54] <marostegui>	 es2014
[07:56:57] <jynus>	 yes
[07:57:03] <jynus>	 we should check query patterns
[07:57:22] <marostegui>	 es2017
[07:57:25] <jynus>	 because last time we had network issues, they didn't fail right away
[07:57:39] <jynus>	 and the thing is- we may need app changes
[07:57:42] <marostegui>	 those two
[07:57:46] <jynus>	 so that they timeout faster
[07:57:52] <jynus>	 or they may create an outage
[07:58:04] <jynus>	 this is not part of the downtime, this is a check I would like to do
[07:58:15] <marostegui>	 Sure, I wasn't aware of those issues
[07:58:27] <jynus>	 so the idea is that if the host is down
[07:58:42] <jynus>	 they are identified quickly and depooled
[07:58:45] <jynus>	 but we suspect
[07:58:52] <jynus>	 (don't know for sure)
[07:59:03] <jynus>	 that if packages are lost (like here)
[07:59:30] <jynus>	 the timeut is so large that it could cause issues due to not being quickly depooled
[07:59:44] <marostegui>	 ah right I see
[07:59:55] <marostegui>	 last time the package lost was really small
[07:59:57] <jynus>	 this would be a good time to check
[07:59:59] <marostegui>	 they were down for seconds
[08:00:01] <marostegui>	 only
[08:00:10] <jynus>	 oh, so maybe not enough
[08:00:18] <marostegui>	 let's see this time :)
[08:36:42] <marostegui>	 https://gerrit.wikimedia.org/r/#/c/362152/
[08:37:07] <jynus>	 oh good
[08:37:13] <marostegui>	 :)
[08:37:18] <jynus>	 remember to deply after stop
[08:37:24] <marostegui>	 yep :)
[08:37:26] <jynus>	 of otherwise you will not be able to
[08:37:54] <jynus>	 good test, btw
[08:38:01] <jynus>	 I only tested on new hosts
[08:47:33] <jynus>	 so, did it work?
[08:49:22] <marostegui>	 just came back from the reboot
[08:49:26] <marostegui>	 so starting mysql now :)
[08:50:09] <marostegui>	 worked like a charm :)
[08:54:54] <jynus>	 icinga, grafana, init.d worked?
[08:56:03] <marostegui>	 looks so :)
[09:00:29] <jynus>	 I have created https://gerrit.wikimedia.org/r/362156 but I am not in a rush to deploy
[09:05:52] <jynus>	 dbproxies complain beacuse some m service has cross-dc availability, right?
[09:06:43] <marostegui>	 I believe so
[09:06:46] <marostegui>	 I am checking
[09:07:35] <marostegui>	 yeah, ie: db2011
[09:07:54] <marostegui>	 which is in row a
[09:08:49] <jynus>	 ok
[09:09:05] <jynus>	 we should probably change that on a multi-instance case
[10:00:43] <mark>	 hey guys
[10:00:49] <mark>	 can we delay the meeting a little bit while we look at the network issues?
[10:01:21] <marostegui>	 fine by me
[10:01:23] <jynus>	 minutes? an hour?
[10:01:32] <mark>	 probably minutes, but no guarantees :)
[10:01:53] <mark>	 if you want to take a break or something we can postpone for a bit longer indeed
[10:02:17] <jynus>	 up to you- if minutes we just hold it, or we can reschedule it
[10:02:21] <jynus>	 12:30 ?
[10:02:24] <mark>	 we should do it today though
[10:02:33] <mark>	 let's try for 12:30, if we don't make that, we'll do it much later today?
[10:02:37] <jynus>	 ok
[10:02:38] <marostegui>	 sure
[10:02:41] <mark>	 cool
[10:08:51] <wikibugs_>	 10DBA, 10Labs, 10User-Urbanecm: Prepare and check storage layer for dinwiki - https://phabricator.wikimedia.org/T169193#3390099 (10Urbanecm)
[10:10:31] <wikibugs_>	 10DBA, 10Labs, 10User-Urbanecm: Prepare and check storage layer for dinwiki - https://phabricator.wikimedia.org/T169193#3390114 (10Marostegui) As the database isn't created in production yet, please let us know when it is done, so we can sanitize it on sanitarium hosts and labs. And create the views.  ``` ro...
[12:19:20] <wikibugs_>	 10DBA: dbstore1001 mysql crashed with: semaphore wait has lasted > 600 seconds - https://phabricator.wikimedia.org/T169050#3390515 (10Marostegui) I have reimported `jawiki.watchlist` from production into dbstore1001, removed the ignore replication filters, enable the normal ones and restarted replication on s6....
[13:05:02] <wikibugs_>	 10Blocked-on-schema-change, 10DBA: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3390636 (10Marostegui)
[13:42:20] <jynus>	 I am going to restart db2072 mariadb several times for systemd tuning
[13:43:41] <marostegui>	 cool
[14:06:35] <wikibugs_>	 10DBA: dbstore1001 mysql crashed with: semaphore wait has lasted > 600 seconds - https://phabricator.wikimedia.org/T169050#3390827 (10Marostegui) 05Open>03Resolved a:03Marostegui I have fixed x1 as well - will close this for now as there is not much we can do. At least the two shards that broke are now fixed.
[14:14:40] <wikibugs_>	 10DBA, 10Patch-For-Review: dbstore2001: s5 thread isn't able to catch up with the master - https://phabricator.wikimedia.org/T168354#3390871 (10Marostegui) An update: The shards keep catching up (as they were really delayed for the maintenance that pretty much lasted a week). x1, s3 and s6 already caught up. T...
[16:16:00] <elukey>	 jynus: re db104[67] replacements - yes we are ok with standard db hosts, ~4TB in raid10 is good enough for our use case.. If this is ok for you I'd ask to Rob to proceed with quotes etc..
[16:16:27] <jynus>	 the thing is
[16:16:34] <jynus>	 this was supposed to be temporary
[16:16:42] <jynus>	 and at some point not needed at all
[16:16:51] <jynus>	 so if we buy a standard config
[16:17:03] <jynus>	 we can use it for other analytics services, too
[16:17:09] <elukey>	 sure sure
[16:17:13] <jynus>	 I am not saying this to take them over
[16:17:30] <jynus>	 also, disk size is standard- less problems with replacements, etc.
[16:17:47] <elukey>	 super
[16:18:00] <jynus>	 for example, we could setup a second dbstore for you and research in the future
[16:18:05] <jynus>	 etc.
[16:18:21] <elukey>	 ah yes if in the future EL data will be entirely on HDFS, right
[16:18:26] <elukey>	 good point
[16:18:35] <jynus>	 or any other thing you need
[16:18:52] <jynus>	 or if you do not need mysqls, we take them and change them for other hard 
[16:19:03] <jynus>	 with more disk or whatever
[16:19:08] <elukey>	 yep yep
[16:19:31] <elukey>	 all right sounds like we are on the same page, will proceed with quotes :)
[16:19:36] <jynus>	 what I would be against is the usage of hds
[16:19:41] <jynus>	 for anything storage
[16:19:44] <jynus>	 that is live
[16:19:53] <jynus>	 your alters would be 10x faster
[16:19:59] <elukey>	 :D
[16:20:09] <jynus>	 it is ok for dead/static data
[18:31:56] <wikibugs_>	 10DBA, 10CirrusSearch, 10Discovery, 10Discovery-Search, and 3 others: archive table needs index starting with timestamp - https://phabricator.wikimedia.org/T164975#3392124 (10Smalyshev) p:05Triage>03Normal @Jcrespo just to be sure, creating test hosts is something you can do? Want to ensure this doesn'...