[04:48:39] <marostegui>	 addshore: I depooled db1092 early in the day yesterday, so maybe that extra load?
[04:48:43] <marostegui>	 but that's worrying if that's the case
[07:09:40] <addshore>	 marostegui: aaah that might be it
[07:09:57] <marostegui>	 addshore: but that's worrying if that is the case
[07:10:14] <addshore>	 Db1126 defintly seemed sad, and sader than the other dB that is the same size still
[07:10:24] <addshore>	 Are they on the same version yet?
[07:10:33] <marostegui>	 yep
[07:10:38] <marostegui>	 only db1111 has a different one
[07:10:45] <marostegui>	 I will repool db1092 today once I am done with the maintenance
[07:11:36] <addshore>	 Yeah, the compression really seems to have a hit on CPU, everything kept ticking but some queries were defintly slow causing some timeouts 
[07:12:28] <addshore>	 I think load was also generally increased over this Easter weekend, I'll look for a graph for that later
[07:12:49] <marostegui>	 we'll have more hosts for s8 anyways
[07:26:22] <marostegui>	 welcome kormat to the Data persistence team!
[07:26:45] <kormat>	 o/
[07:27:46] <jynus>	 hello, kormat!
[07:28:07] <jynus>	 should we have a welcome hangout?
[07:28:11] <marostegui>	 kormat: jynus is the other member of the team, jaime :-)
[07:28:33] <marostegui>	 jynus: let us get the onboarding a bit advanced and we can do it later yeah :)
[07:29:00] <jynus>	 ok, just wanted to say hello, hello kormat
[07:29:31] <kormat>	 jynus: hi :) i hear you're less of a morning person than marostegui, so i prefer you already ;)
[07:29:38] <marostegui>	 :_(
[07:29:51] <marostegui>	 that hurts
[07:30:36] <jynus>	 marostegui: don't worry, our children love us both equally :-DDDD
[07:30:42] <marostegui>	 hahaha
[07:31:17] <jynus>	 https://jynus.com/gif/gerrit_plus_1.gifv
[07:31:26] <marostegui>	 addshore: Once I have repooled db1092 we can depool another host and do the first drop of the wb_terms if you like
[07:31:27] <marostegui>	 Amir1: ^
[07:31:39] <Amir1>	 Sure!
[07:31:41] <Amir1>	 Thanks
[07:32:01] <marostegui>	 Amir1 and addshore please meet kormat too!
[07:32:18] <Amir1>	 kormat: Hey o/
[07:32:35] <Amir1>	 (We are engineers at wikidata team, Wikimedia Germany)
[07:32:38] <marostegui>	 kormat: Amir1 and addshore are WMDE (wikimedia deutschland) developers that work with us a lot - we'll schedule a meeting with them in a few days to say hello if you want
[07:32:59] <kormat>	 oh excellent. Guten Tag :)
[07:33:07] <addshore>	 Hi kormat ! :D
[07:33:19] <Amir1>	 None of us are German, I can order a beer in German though 
[07:33:35] <kormat>	 i live in .ch, but i'm not a german-speaker either :)
[07:33:36] <marostegui>	 I see you have been advancing with your german lessons...
[07:33:40] <kormat>	 haha
[07:34:27] <addshore>	 https://usercontent.irccloud-cdn.com/file/gnlJy3zw/image.png
[07:34:34] <addshore>	 marostegui: ^^ that was the graph I was going to send you
[07:34:46] <addshore>	 the big spike was last ngiht, but note the spike in the 2/3 days running up to it too
[07:34:58] <marostegui>	 addshore: that is the easter one, no?
[07:35:01] <addshore>	 (thats showing wikidata write modules)
[07:35:02] <addshore>	 yes
[07:35:38] <addshore>	 https://usercontent.irccloud-cdn.com/file/0W3VoNUH/image.png
[07:36:01] <addshore>	 looks like yesterday db1111 ended up getting less load and db1126 more while the other server was depooled?
[07:36:05] <marostegui>	 addshore: I am worried that depooling a single slave on wikidata causes so much load, but looks like that wasn't the cause anyways
[07:36:25] <marostegui>	 addshore: yeah, db1111 was moved into API (to replace db1092)
[07:36:37] <marostegui>	 addshore: db1092 should be done by today or tomorrow 
[07:36:48] <marostegui>	 db1111 is the one using 10.4 which better CPU-wise
[07:36:58] <addshore>	 aaah and the api ends up resulting in less load?
[07:37:09] <marostegui>	 but we are going to get extra hosts in s8 "soon"
[07:37:13] <marostegui>	 addshore: yep
[07:37:40] <addshore>	 interesting that db1111 was in the api, and the api saw a bit of a performance hit, but the load on db1111 was lower.
[07:37:58] <marostegui>	 keep in mind that db1111 got less main traffic load
[07:38:03] <marostegui>	 let me see if I can show you the diff
[07:38:29] <marostegui>	 https://phabricator.wikimedia.org/P10963
[07:38:42] <marostegui>	 so db1092's load was spread across the others
[07:39:00] <marostegui>	 we need to move more hosts to 10.4 I reckon
[07:39:02] <marostegui>	 :)
[07:43:14] <addshore>	 Interesting, I would have thought that we would get more api load than anything else :P
[07:43:26] <marostegui>	 we might, but it doesn't look as heavy
[07:44:42] <addshore>	 Yeah, im wondering which queries are really "taking the mustard" and we might want to fix -- https://en.wikipedia.org/wiki/Taking_the_piss
[08:28:43] <jynus>	 is there a ticket for kormat's onboarding?
[08:29:10] <marostegui>	 https://phabricator.wikimedia.org/T250134
[08:29:51] <jynus>	 thanks
[08:58:54] <jynus>	 I will be working on the calendar event, but may need some answers
[09:00:56] <godog>	 hi! I'm going through clinic duty tasks, fair to say this task is more about test db access than analytics access?
[09:00:59] <godog>	 https://phabricator.wikimedia.org/T249059
[09:01:27] <marostegui>	 godog: hola, yeah, check this: https://phabricator.wikimedia.org/T249059#6042209
[09:01:36] <marostegui>	 godog: I am waiting for them to create the task 
[09:01:45] <jynus>	 manuel is to say, but I would decline it when a new task is opened
[09:01:56] <marostegui>	 +1
[09:02:04] <jynus>	 you can set is as stalled, maybe?
[09:02:09] <jynus>	 for now
[09:02:20] <jynus>	 and promting them to do that again?
[09:02:30] <godog>	 sweet, yeah I'll do that now, thanks marostegui and jynus 
[09:05:10] <marostegui>	 thanks
[09:37:00] <jynus>	 marostegui: can I delete dbprov1001:/srv/backups/dumps/latest/emergency ?
[09:38:53] <marostegui>	 jynus: yes please
[09:38:54] <marostegui>	 thanks
[11:50:11] <Amir1>	 marostegui: Should I run the script again for drifts?
[12:00:26] <marostegui>	 Amir1: let me finish a couple of more tasks to get them out of the way
[12:06:21] <Amir1>	 Thanks! no rush
[12:13:58] <marostegui>	 jynus: for whenever you are bored https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/587628/
[12:21:14] <wikibugs>	 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 7 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Marostegui) Can this task be closed is th...
[12:21:39] <wikibugs>	 10DBA: decommission dbproxy1011.eqiad.wmnet - https://phabricator.wikimedia.org/T249590 (10Marostegui)
[12:22:29] <wikibugs>	 10DBA: decommission dbproxy1011.eqiad.wmnet - https://phabricator.wikimedia.org/T249590 (10Marostegui) Grants for `haproxy` user were dropped at: T231280#6051126
[12:22:55] <wikibugs>	 10DBA, 10Data-Services, 10Operations: Replace labsdb (wikireplicas) dbproxies: dbproxy1010 and dbproxy1011 - https://phabricator.wikimedia.org/T231520 (10Marostegui)
[12:24:48] <wikibugs>	 10DBA, 10Data-Services, 10Operations: Replace labsdb (wikireplicas) dbproxies: dbproxy1010 and dbproxy1011 - https://phabricator.wikimedia.org/T231520 (10Marostegui) 05Open→03Resolved dbproxy1019 has been working fine for a week. Considering this done. Next step is to decommission dbproxy1011 : T249590
[12:24:51] <wikibugs>	 10DBA: Remove grants for the old dbproxy hosts from the misc databases - https://phabricator.wikimedia.org/T231280 (10Marostegui)
[15:18:25] <wikibugs>	 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 7 others: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views - https://phabricator.wikimedia.org/T249565 (10Jdforrester-WMF) 05Open→03Resolved Ye...
[19:54:06] <wikibugs>	 10DBA, 10Operations: move db1114 to s8 - https://phabricator.wikimedia.org/T250224 (10CDanis)
[19:55:44] <marostegui>	 thanks cdanis <3
[19:55:53] <cdanis>	 <3
[19:57:04] <wikibugs>	 10DBA, 10Core Platform Team: text table still has old_* fields and indexes on some hosts - https://phabricator.wikimedia.org/T250066 (10daniel) Afaik Amir is correct. pinging @tstarling to confirm.  I'd be very surprised if anything still used these fields. But I have had such surprises before :P
[19:59:14] <wikibugs>	 10DBA, 10Core Platform Team, 10MediaWiki-User-management, 10Schema-change: Rename ipb_address index on ipb_address to ipb_address_unique - https://phabricator.wikimedia.org/T250071 (10daniel) Codesearch shows some references to the index in updater code, but none in production code or extensions. We should...
[20:22:11] <wikibugs>	 10DBA, 10Gerrit: Investigate Gerrit troubles to reach the MariaDB database - https://phabricator.wikimedia.org/T247591 (10hashar) >>! In T247591#5975312, @hashar wrote: > I have checked logstash again, the issue has not occurred since March 11.  Checked again, we "just" had a spike of 26 errors between 2020-03...
[20:26:22] <wikibugs>	 10DBA, 10MediaWiki-User-management, 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Rename ipb_address index on ipb_address to ipb_address_unique - https://phabricator.wikimedia.org/T250071 (10daniel)
[22:57:03] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar), 10codfw-rollout: [RFC] improve parsercache replication, sharding and HA - https://phabricator.wikimedia.org/T133523 (10Krinkle)
[23:13:50] <wikibugs>	 10DBA, 10MediaWiki-Parser, 10Operations, 10Datacenter-Switchover-2018, and 3 others: parsercache used disk space increase - https://phabricator.wikimedia.org/T206740 (10Krinkle)