[05:19:16] <wikibugs>	 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad, 10User-Banyek: db1069 has errored disk in slot 7 - https://phabricator.wikimedia.org/T205253 (10Marostegui)
[06:14:05] <wikibugs>	 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Wikidata-Campsite, 10User-Ladsgroup: Schema change for adding indexes of ct_tag_id - https://phabricator.wikimedia.org/T203709 (10Marostegui)
[07:22:13] <wikibugs>	 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad, 10User-Banyek: db1069 has errored disk in slot 7 - https://phabricator.wikimedia.org/T205253 (10Banyek) 05Open>03Resolved The rebuild finished:  Adapter 0 -- Virtual Drive Information: Virtual Drive: 0 (Target Id: 0) Name                : RAID Level...
[07:33:52] <banyek>	 marostegui: I seen your comment about dbstore1002, my next designated table to convert is wikishared/cx_corpora.ibd 
[07:34:02] <banyek>	 can I do it, or your alters affecting that?
[07:34:26] <marostegui>	 banyek: They are not affecting it, but I want to minimize large writing operations on that host
[07:34:37] <marostegui>	 So, can you give me a couple of hours?
[07:34:44] <banyek>	 sure I can
[07:35:00] <marostegui>	 btw, I think we should not leave dbstore2002:s2 replication stopped for the weekend
[07:35:07] <marostegui>	 I am fine leaving it running the alters, but with replication flowing
[07:35:54] <banyek>	 there's 3 tables still to go - and the largest ones
[07:36:22] <banyek>	 I'd say I'd let it fun for today, and at the end of the day I stop the conversion, and start replication
[07:36:35] <marostegui>	 Sure, I am fine with leaving the alters for the weekend, but maybe we should start replication too, so it can catch up slowly
[07:36:47] <banyek>	 ok
[07:37:02] <banyek>	 I stop the alter/start the replication now, and you can run the alters
[07:37:22] <banyek>	 from monday to wednesday the compression will probably finsh
[07:37:23] <marostegui>	 what?
[07:37:38] <marostegui>	 My stuff is on dbstore1002, not dbstore2002
[07:37:55] <banyek>	 "btw, I think we should not leave dbstore2002:s2 replication stopped for the weekend"
[07:38:01] <banyek>	 I thought we are talking about it
[07:38:08] <marostegui>	 ˜/marostegui 9:35> btw, I think we should not leave dbstore2002:s2 replication stopped for the weekend"
[07:38:11] <marostegui>	 dbstore2002
[07:38:17] <marostegui>	 but dbstore1002 :)
[07:38:21] <marostegui>	 I am only touching dbstore1002
[07:38:50] <marostegui>	 So my thoughts are: dbstore1002 -> give me some hours to finish my mainteanance
[07:38:58] <marostegui>	 dbstore2002 -> leave alters running but start replication
[07:40:33] <banyek>	 ah. ok
[07:40:50] <marostegui>	 I am also fine with stopping alters on dbstore2002
[07:40:54] <marostegui>	 and resume on monday
[07:40:55] <marostegui>	 up to you
[07:43:11] <banyek>	 I'd say I continue compressing it today, stop the compression for the weekend and resumre replication once it caught up I resume compression even it's monday, or earlier. The s2 would be nice to finish until the next backup
[07:43:41] <marostegui>	 sounds good
[07:44:18] <marostegui>	 I think I will be done with dbstore1002 in 1h or so
[07:44:38] <marostegui>	 I will ping you
[07:45:03] <banyek>	 perfect, thanks!
[08:07:42] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: Issues with mgmt interface on es2001 host - https://phabricator.wikimedia.org/T204928 (10Banyek) I had experiences with some Sun Fire servers, their mgmt interfaces were constantly broken, but mostly a powercycle solved it.  I don't know if we can afford to fully power down...
[08:09:39] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: Issues with mgmt interface on es2001 host - https://phabricator.wikimedia.org/T204928 (10jcrespo) @Banyek, please read https://wikitech.wikimedia.org/wiki/Management_Interfaces
[08:13:00] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: Issues with mgmt interface on es2001 host - https://phabricator.wikimedia.org/T204928 (10Banyek) a:03Banyek I can try these
[08:16:44] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: Issues with mgmt interface on es2001 host - https://phabricator.wikimedia.org/T204928 (10jcrespo) Don't, this requires a power drain, you cannot help with this.
[08:16:53] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: Issues with mgmt interface on es2001 host - https://phabricator.wikimedia.org/T204928 (10jcrespo) a:05Banyek>03None
[08:34:38] <wikibugs>	 10DBA, 10Epic, 10Wikimedia-Incident: Improve regular production database backups handling - https://phabricator.wikimedia.org/T138562 (10jcrespo)
[08:34:40] <wikibugs>	 10DBA, 10Patch-For-Review: Finish eqiad metadata database backup setup (s1-s8, x1) - https://phabricator.wikimedia.org/T201392 (10jcrespo) 05stalled>03Open
[08:35:13] <wikibugs>	 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Wikidata-Campsite, 10User-Ladsgroup: Schema change for adding indexes of ct_tag_id - https://phabricator.wikimedia.org/T203709 (10Marostegui)
[08:36:02] <wikibugs>	 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Wikidata-Campsite, 10User-Ladsgroup: Schema change for adding indexes of ct_tag_id - https://phabricator.wikimedia.org/T203709 (10Marostegui) This is all done in eqiad. Once we are back in eqiad this will be deployed on codfw...
[08:45:18] <marostegui>	 banyek: you can go ahead with dbstore1002
[08:45:41] <banyek>	 thanks
[09:07:25] <wikibugs>	 10DBA, 10Data-Services, 10Datasets-General-or-Unknown: Archive and drop education program (ep_*) tables on all wikis - https://phabricator.wikimedia.org/T174802 (10Marostegui) What is pending here?  1) Disable the extension everywhere? 2) Get a list of wikis where it is/was disabled? 3) Do the proper table D...
[09:09:41] <banyek>	 I learned that dbstore1002 is doing multi source replication, but I don't know how to decide which database is from which source. I'd like to convert to TokuDB the table 'wikishared.cx_corpora, but I am not sure which replication monitor on icinga should I downtime
[09:10:47] <banyek>	 (except looking for that db in all the clusters until I find it, but I am pretty sure that's theres's a more professional way to do it)
[09:11:45] <marostegui>	 You have the dblist files on the mediawiki-config repo, which have a list of dbs per section
[09:12:09] <banyek>	 ok
[09:13:22] <jynus>	 you have a summary on the db-eqiad.php and db-codfw.php configs, too
[09:13:46] <jynus>	 or at https://noc.wikimedia.org/db.php , banyek
[09:13:56] <jynus>	 somtimes a browser is nice to have it on a separate window
[09:14:08] <marostegui>	 interesting, that one doesn't contain x1
[09:14:24] <jynus>	 ?
[09:14:45] <jynus>	 x1 contains all?
[09:14:46] <marostegui>	 There is no x1 on https://noc.wikimedia.org/db.php
[09:15:05] <jynus>	 but x1 is not a separate "cluster"
[09:15:15] <jynus>	 it contains data from all clusters
[09:15:34] <marostegui>	 Yeah, but it has its own separate set of servers
[09:15:47] <jynus>	 sure, I am using here mediawiki definition of cluster
[09:15:54] <jynus>	 not real one
[09:15:55] <marostegui>	 Ah ok ok :)
[09:16:25] <jynus>	 tehnically , I think x1 is a cluster, but not a section
[09:16:38] <jynus>	 but the names are used very loosly
[09:16:48] <marostegui>	 Yeah, I do consider x1 as a section, because it is separated
[09:16:58] <marostegui>	 But yeah, depends on what you consider a section 
[09:17:02] <jynus>	 from our perspective (DBAs) I would call cluster to all sets of hosts with the same data
[09:17:25] <marostegui>	 then x1 should be a cluster appearing on that page, as it lists clusters
[09:17:28] <marostegui>	 haha
[09:17:31] <jynus>	 but it ambiguous because you can also call all s* hosts a cluster, just partitioned by wiki
[09:18:07] <jynus>	 I think the main difference is the logical classification vs the functional one
[09:18:22] <jynus>	 as in dataset classification, where s1 core and labsdb have the same data
[09:18:39] <jynus>	 but s1 core and labs are used for different applications
[09:18:50] <jynus>	 and that leads to confusion
[09:19:08] <jynus>	 we could start using non-ambiguous names like "replica set"
[09:19:14] <marostegui>	 haha yeah
[09:19:28] <marostegui>	 so we would have section, shard, cluster, replica set and then mix all of them!
[09:19:30] <jynus>	 x1 is a replica set
[09:19:44] <jynus>	 but contains data from all sections
[09:20:01] <jynus>	 that are different from data in sections of the core metadata
[09:20:15] <jynus>	 but it is functionaly part of production metadata set of servers
[09:20:28] <jynus>	 all very clear
[09:20:45] <jynus>	 ^banyek if you understood all that, you are already a wmf dba
[09:21:28] <jynus>	 we could change dbtree/tendril into a better diagram
[09:22:01] <jynus>	 which clasifies things by proximity and by color, for example, to show the 2 dimensinal classification
[09:22:57] <banyek>	 https://kep.cdn.index.hu/1/0/703/7030/70307/7030771_c1da4eed1d2de24caf607b4699a27b77_wm.gif
[09:24:22] <banyek>	 the fun part is that I can't find either the 'wikishared' nor the 'cx_corpora' strings in that directory
[09:24:25] <marostegui>	 then what about parsercache? what are those? or es?
[09:24:51] <marostegui>	 banyek: maybe wikiSHARE can give you a hint, with all the stuff we discussed above :)
[09:24:53] <jynus>	 ywah
[09:25:01] <jynus>	 those are not wikis
[09:25:11] <jynus>	 so I think those are hardcoded
[09:26:48] <banyek>	 I'd say X1 one, but I am trying to not step on a booby trap
[09:27:00] <marostegui>	 go and check
[09:27:08] <marostegui>	 :)
[09:27:18] <jynus>	 banyek: I may send you purposedly bad patches to review so you can caught them and vote -1 to them
[09:28:04] <jynus>	 (they won't really be on purpose, but I would claim it was a "training" execice when I do something stupid
[09:28:16] <banyek>	 ^^ It's almost the same when my son tells me that 'Let's play hide and seek, you'll find me under the bed' :-P
[09:28:37] <jynus>	 banyek: well, not if only some will be bad and some will
[09:28:48] <jynus>	 not
[09:28:49] <banyek>	 true
[09:30:06] <banyek>	 it's x1 indeed 
[09:30:25] <banyek>	 (Holly molly, 963 rows as result of 'SHOW DATABASES')
[09:30:26] <jynus>	 don't think the more you know, the less mistakes you will do- it is actually the oppsite
[09:30:47] <jynus>	 banyek: yep, don't ever use information_schema.tables on x1 or s3
[09:31:07] <jynus>	 that is why we tar those databases, remember what I showed you yesterday?
[09:31:20] <banyek>	 yes
[09:32:30] <jynus>	 https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/profile/templates/mariadb/backups-codfw.cnf.erb;3ea0aee91b78f277b0dbe6d0a0a9fa76bb67d0c5?as=source&blame=off
[09:32:36] <jynus>	 ^see the archive: True
[09:32:43] <jynus>	 which basically means lots of stuff
[09:33:38] <jynus>	 for even more context, there is also pending https://phabricator.wikimedia.org/T184805
[09:33:59] <jynus>	 which BTW, I wanted to talk to you marostegui (not now, but will throw this to you)
[09:34:07] <jynus>	 (on monday)
[09:34:29] <marostegui>	 sure 
[09:34:49] <jynus>	 if we are cool on maintenance, and we have 1 week left, we could even think to prepare that or to allow some of the blocked mediawiki tasks
[09:35:56] <marostegui>	 yeah, we'd need to see what cloud say
[09:35:58] <marostegui>	 about moving it
[09:36:04] <marostegui>	 (+1 from my side to move it)
[09:36:05] <jynus>	 cloud?
[09:36:16] <jynus>	 I am not that worried about labswiki
[09:36:29] <marostegui>	 ah, you are talking about the other ones?
[09:36:30] <jynus>	 in fact, I think we cannot do that
[09:36:36] <jynus>	 yep
[09:36:43] <jynus>	 I know there is not a lot of time
[09:36:50] <jynus>	 and there are many reasons to say no
[09:37:04] <jynus>	 (so for now please think it we will talk on monday)
[09:37:10] <marostegui>	 yep
[09:37:12] <jynus>	 let me just say 1 reason to say ues
[09:37:26] <jynus>	 it will be much much harder with replication later on
[09:37:39] <marostegui>	 No need to convince me, I am all forward to move those
[09:37:48] <marostegui>	 We are "wasting" hardware on s5, that is also a reason
[09:37:53] <jynus>	 and we can do it like with s8, with alway a way to rollback
[09:37:56] <marostegui>	 So let's chat about it on monday
[09:38:08] <jynus>	 yep
[09:40:20] <jynus>	 just one last think- wasting is bad, s3 reaching its limits is worse
[09:40:23] <jynus>	 *thing
[09:41:20] <jynus>	 both due to the amount of objects and the lack of partitioning non really-no-longer-small-wikis
[09:41:32] <jynus>	 *on
[09:41:50] <marostegui>	 https://phabricator.wikimedia.org/P7601
[09:41:57] <marostegui>	 cebwiki…jeeeez
[09:42:10] <marostegui>	 haha templatelinks is 414G
[09:42:37] <jynus>	 yeah, it is bit missleading
[09:43:02] <jynus>	 but the same problem, no matter what
[09:43:07] <marostegui>	 yep
[09:44:15] <jynus>	 enwikivoyage has a higher load than the size would suggest also
[10:20:02] <banyek>	 rearding this ticket: https://phabricator.wikimedia.org/T172490
[10:20:15] <banyek>	 arent' we considering to try out PMM ?
[10:21:52] <jynus>	 you are so obsessed with PMM
[10:22:01] <jynus>	 we already have swap monitoring on prometheus
[10:22:25] <jynus>	 and better suited metrics
[10:22:30] <jynus>	 that ticket is about alerting
[10:23:06] <jynus>	 PMM is a marketing scam
[10:23:06] <banyek>	 No, not at all. If I am obsessed with monitoring, that's vividcortex, but I won't ever say it out loud :)
[10:23:59] <jynus>	 it is just a docker container with prometheus and the mysql plugins- which we already have setup on bare metal
[10:24:12] <jynus>	 so in a way, we are already using PMM
[10:24:35] <jynus>	 but no software is magical- it needs tuning for our needs
[10:24:46] <jynus>	 that is what that ticket is about
[10:24:57] <jynus>	 our main alerting software, at the moment is icinga
[10:25:27] <jynus>	 we need eiter a bridge from icinga to prometheus, an implement it or a separate thing, on of the 2
[10:26:02] <banyek>	 I see.
[10:26:05] <jynus>	 remember 1) I attended the same conferences than you
[10:26:12] <jynus>	 and 2) I worked for percona
[10:27:21] <jynus>	 not that PMM is bad, but our problem is not a lack of technology stuck, but to adapt it and scale it to our size
[10:32:28] <banyek>	 I never seen a conference talk about PMM, because i was really satisfied with vividcortex :)
[10:33:37] <banyek>	 but they're really expensive in one hand, and in other: they are selling the service, not self hosted, that's why I never mentioned them
[10:33:49] <jynus>	 banyek: http://smalldatum.blogspot.com/2016/09/excited-about-percona-live-amsterdam.html
[10:34:03] <jynus>	 this 1000 times
[10:35:53] <banyek>	 I have no problems with 'Why aren't you using Y?' but I agree that 'You should be using Z!' is bad
[10:36:04] <jynus>	 I do have problems with both
[10:36:48] <jynus>	 I am tired of getting asked "Why aren't you using postgres" or similar when the other person doesn't even know what my environement is or what I am trying to do
[10:36:52] <banyek>	 'Why aren't you using PMM?' 'Because it's a marketing scam, it doesn't know more than Graphana with MySQL plugins' -> TIL
[10:37:53] <jynus>	 "it doesn't know more than Graphana with MySQL plugins" no
[10:38:26] <jynus>	 PMM IS prometheus mysqld exporter + prometheus + graphana , which we are already using
[10:39:13] <banyek>	 I always liked learned this way - becuause _of the explanation_ of "why not" can carry context.
[10:40:06] <jynus>	 I beg you to ask what are we using, what are our problems before
[10:40:25] <jynus>	 that is an interesting question
[10:40:46] <jynus>	 or note gaps on our monitoring
[10:40:50] <jynus>	 conceptually
[10:41:21] <jynus>	 another typical question is "why are we using mariadb?"
[10:41:54] <jynus>	 but people don't ask that, they ask "why are not using $other"
[10:42:18] <_joe_>	 jynus: I was wondering why don't we use Postgres?
[10:42:23] <_joe_>	 there see
[10:42:26] <_joe_>	 I just did!
[10:43:35] <_joe_>	 tbh, I think the question per-se is innocuous, if asked with genuine interest and respect
[10:43:56] <_joe_>	 I might ask what are the shortcomings of X (which I am using) that make it unsuitable to you
[10:44:01] <jynus>	 well, it is not as much as the question itself
[10:44:06] <_joe_>	 the issue is almost all nerds think they know better
[10:44:11] <jynus>	 exactly
[10:44:24] <_joe_>	 so the problem is not the question, but the attitude
[10:44:25] <jynus>	 it is the assumption of knowing better
[10:44:27] <jynus>	 yers
[10:44:47] <jynus>	 that is why framing the question in another way may help with the attitude
[10:45:08] <banyek>	 For me the question 'why are we using mariadb?' is answerable with the exact same answer as 'why are not using $other' like 'Because MariaDB is ... and $other is not and we need Z.'
[10:45:10] <_joe_>	 he who's innocent can throw the first stone ;)
[10:45:33] <jynus>	 I don't even have issues with the questions
[10:45:47] <jynus>	 but after getting so many, it shows bad on you
[10:46:00] <jynus>	 you == the person who ask it
[10:46:29] <jynus>	 so I am not saying "don' ask that"
[10:47:10] <jynus>	 I do other worse things in other fields
[10:47:17] <_joe_>	 ok, I don't think this is a congregation of cargo-culters and db fanbois here, right?
[10:47:25] <_joe_>	 so we can assume good faith 
[10:47:49] <jynus>	 _joe_: most conferences are :-)
[10:47:52] <_joe_>	 oh yes
[10:48:03] <_joe_>	 I thought the discussion was tied to our group
[10:48:11] <jynus>	 let me give you an equivalent question
[10:48:26] <_joe_>	 why don't we use nodejs for mediawiki
[10:48:30] <jynus>	 what is your immediate reaction if I asked, "why don't you use nodejs?"
[10:48:45] <jynus>	 I guess a sight
[10:48:51] <_joe_>	 it's not like that question wasn't asked withing the WMF :P
[10:49:07] <jynus>	 a polite answer that doesn't fully tell the whole picture
[10:49:23] <jynus>	 and internally, you try to distance from the person to ask taht question
[10:49:40] <jynus>	 (maybe, that is what I woul do)
[10:50:57] <banyek>	 well, anyhow, I go to have some lunch soon :)
[10:51:50] <banyek>	 btw converting cx_corpora finished, I go forward to dewiki.flaggedtemplates
[10:53:37] <banyek>	 x1 caught up
[10:59:56] <wikibugs>	 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Wikidata-Campsite, 10User-Ladsgroup: Schema change for adding indexes of ct_tag_id - https://phabricator.wikimedia.org/T203709 (10Ladsgroup) <3 <3 <3
[11:08:30] <wikibugs>	 10DBA, 10Growth-Team, 10StructuredDiscussions, 10MW-1.27-release (WMF-deploy-2015-12-08_(1.27.0-wmf.8)), 10Patch-For-Review: Cleanup ptwikibooks conversion - https://phabricator.wikimedia.org/T119509 (10Trizek-WMF)
[13:35:10] <jynus>	 we are going to be a bit low on space at db1116 for some time: https://grafana.wikimedia.org/dashboard/file/server-board.json?refresh=1m&panelId=17&fullscreen&orgId=1&var-server=db1116&var-network=eno1&from=1538120080526&to=1538141680526
[13:35:28] <jynus>	 so you don't worry when we get a silenced alert
[13:38:37] <marostegui>	 thanks for the heads up
[13:40:49] <jynus>	 we ended up at 91%
[14:02:07] <jynus>	 the s8 and m3 backups on codfw seem stuck, but the alert didn't work
[14:03:30] <marostegui>	 what's the check returning?
[14:03:35] <marostegui>	 like, if you run it manually I mean
[14:03:49] <jynus>	 Backup for s8 at codfw taken less than 8 days ago and larger than 10 GB: Last one 2018-09-26 04:10:45 from dbstore2001.codfw.wmnet:3318 (99 GB)
[14:04:03] <jynus>	 which is right, that backup completed
[14:04:45] <jynus>	 but on the metadata it says ongoing (maybe the database was restarted at that time)
[14:05:01] <jynus>	 actually, I see the issue
[14:05:06] <jynus>	 there are 2 entries
[14:05:14] <jynus>	 the right one, so the alart works
[14:05:22] <jynus>	 166 | dump.s8.2018-09-26--04-10-44      | finished | dbstore2001.codfw.wmnet:3318
[14:05:37] <jynus>	 172 | dump.s8.2018-09-26--12-07-07      | ongoing  | dbstore2001.codfw.wmnet:3318 
[14:05:42] <jynus>	 I know what it is
[14:05:56] <jynus>	 I accidentally run the dump process and killed it
[14:06:02] <jynus>	 and it show it on the metadata
[14:06:10] <marostegui>	 Aha!
[14:06:13] <marostegui>	 Problem solved :)
[14:06:14] <jynus>	 so actually things are working as expected- including an aborted run
[14:06:40] <jynus>	 so if you just run dump_section.py with no parameters
[14:06:47] <jynus>	 remember you are doing a full dump
[14:06:52] <jynus>	 (I didn't)
[14:07:10] <marostegui>	 maybe it should include a warning and a y/n ?
[14:07:17] <marostegui>	 like: you want to continue with a full dump?
[14:07:35] <jynus>	 that doesn't feel right with a cron
[14:07:54] <marostegui>	 does the cron calls it without paramenters?
[14:07:56] <jynus>	 but maybe an extra parameter with
[14:08:14] <jynus>	 --cron
[14:08:21] <jynus>	 or --unsuperviced
[14:08:30] <marostegui>	 yeah, something like that
[14:09:03] <jynus>	 or make --config compulsory
[14:09:19] <jynus>	 and otherwise try to do a local dump, which would fail immediately
[14:15:14] <wikibugs>	 10DBA, 10Cloud-Services, 10User-Urbanecm: Prepare and check storage layer for liwikinews - https://phabricator.wikimedia.org/T205713 (10Urbanecm)
[14:16:23] <marostegui>	 banyek: ^ you want to me to explain to you on monday how to handle those tickets?
[14:17:24] <banyek>	 Yes please!
[14:17:58] <marostegui>	 banyek: this is a similar ticket: https://phabricator.wikimedia.org/T167715
[14:18:03] <wikibugs>	 10DBA, 10Cloud-Services, 10User-Urbanecm: Prepare and check storage layer for yuewiktionary - https://phabricator.wikimedia.org/T205714 (10Urbanecm)
[14:22:32] <wikibugs>	 10DBA, 10Patch-For-Review: Finish eqiad metadata database backup setup (s1-s8, x1) - https://phabricator.wikimedia.org/T201392 (10jcrespo) ``` root@neodymium:~/tendril/bin$ host=db1116.eqiad.wmnet; port=3317; bash tendril-host-add.sh $host $port ~/.my.cnf.tendril tendril | mysql -h db1115.eqiad.wmnet tendril &...
[14:24:54] <jynus>	 so I have added a new s7 and s8 instance, FYI in case that creates issues with ongoing schema changes
[14:26:17] <marostegui>	 thanks, no schema changes scheduled for s7 or s8, those are done
[14:27:18] <jynus>	 they should be now on ./section and tendril
[14:28:03] <jynus>	 I will now set them up on backups, but may not be ready fully as they need compression
[14:29:06] <jynus>	 The plan is to setup s6 and x1 on dbstore1001
[14:29:24] <marostegui>	 dbstore1001 has s1 and s4 now?
[14:29:26] <marostegui>	 or which ones?
[14:29:39] <jynus>	 s1 only
[14:29:48] <jynus>	 but it has the local backups
[14:30:13] <jynus>	 we would need 1 extra host for backup source + local stores new
[14:30:18] <jynus>	 at the very least
[14:30:26] <jynus>	 plust the 2 testing hosts
[14:30:34] <jynus>	 as those have been used for backup sources
[14:31:00] <jynus>	 the local stores with 10Gb ethernet
[14:31:24] <jynus>	 much more for codfw, where everthing is old
[14:33:07] <wikibugs>	 10DBA, 10Cloud-Services, 10User-Banyek, 10User-Urbanecm: Prepare and check storage layer for yuewiktionary - https://phabricator.wikimedia.org/T205714 (10Banyek)
[14:33:14] <wikibugs>	 10DBA, 10Cloud-Services, 10User-Banyek, 10User-Urbanecm: Prepare and check storage layer for liwikinews - https://phabricator.wikimedia.org/T205713 (10Banyek)
[14:34:21] <marostegui>	 jynus: yeah, we need to review what's pending/next to purchase for backups
[14:35:50] <wikibugs>	 10DBA, 10Cloud-Services, 10User-Banyek, 10User-Urbanecm: Prepare and check storage layer for yuewiktionary - https://phabricator.wikimedia.org/T205714 (10Banyek) Please ping me (or anybody in the persistance team) when the wiki is created, and I can do the sanitzation
[14:35:55] <wikibugs>	 10DBA, 10Cloud-Services, 10User-Banyek, 10User-Urbanecm: Prepare and check storage layer for liwikinews - https://phabricator.wikimedia.org/T205713 (10Banyek) Please ping me (or anybody in the persistance team) when the wiki is created, and I can do the sanitzation
[14:36:46] <jynus>	 did you tell him that if wikis would be private, the workflow would be different?
[14:36:53] <marostegui>	 yes
[14:36:56] <jynus>	 cool
[14:39:16] <wikibugs>	 10DBA, 10Cloud-Services, 10User-Banyek, 10User-Urbanecm: Prepare and check storage layer for yuewiktionary - https://phabricator.wikimedia.org/T205714 (10Banyek) p:05Triage>03Normal
[14:39:23] <wikibugs>	 10DBA, 10Cloud-Services, 10User-Banyek, 10User-Urbanecm: Prepare and check storage layer for liwikinews - https://phabricator.wikimedia.org/T205713 (10Banyek) p:05Triage>03Normal
[14:39:29] <wikibugs>	 10DBA, 10Patch-For-Review: Finish eqiad metadata database backup setup (s1-s8, x1) - https://phabricator.wikimedia.org/T201392 (10jcrespo)
[14:40:14] <marostegui>	 jynus: should we consider https://phabricator.wikimedia.org/T196376 resolved?
[14:41:42] <jynus>	 I think yes?
[14:41:46] <jynus>	 we may change their roles
[14:41:51] <jynus>	 but we close the one on codfw
[14:41:55] <jynus>	 *closed
[14:42:01] <jynus>	 when they had at least some usse
[14:42:08] <marostegui>	 yeah, let's close that one too
[14:42:38] <wikibugs>	 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy - https://phabricator.wikimedia.org/T190704 (10jcrespo)
[14:42:41] <wikibugs>	 10DBA, 10Patch-For-Review: Productionize old/temporary eqiad sanitariums - https://phabricator.wikimedia.org/T196376 (10jcrespo) 05Open>03Resolved a:05jcrespo>03Marostegui All hosts setup and in production usage.
[14:43:06] <wikibugs>	 10DBA, 10Patch-For-Review: Productionize old/temporary eqiad sanitariums - https://phabricator.wikimedia.org/T196376 (10jcrespo)
[14:43:09] <wikibugs>	 10DBA, 10Patch-For-Review: Productionize old/temporary eqiad sanitariums - https://phabricator.wikimedia.org/T196376 (10Marostegui)
[14:43:31] <marostegui>	 We did exactly the same update
[14:43:40] <marostegui>	 haha
[14:43:41] <jynus>	 :-) https://phabricator.wikimedia.org/T196376#4622120
[14:43:48] <jynus>	 sadly no conflic reslution on phab
[14:43:48] <marostegui>	 gah! you won!
[14:43:50] <jynus>	 unlike wiki
[14:43:54] <marostegui>	 yeah
[14:43:57] <marostegui>	 the last one wins XD
[14:44:35] <wikibugs>	 10DBA, 10Patch-For-Review: Productionize old/temporary eqiad sanitariums - https://phabricator.wikimedia.org/T196376 (10Marostegui)
[14:44:37] <jynus>	 what is missing from T159423 dbstore1002 and labsdb?
[14:44:38] <stashbot>	 T159423: Meta ticket: Migrate multi-source database hosts to multi-instance - https://phabricator.wikimedia.org/T159423
[14:45:05] <marostegui>	 yeah, it was mostly a discussion ticket that turned into a tracking one
[14:45:06] <jynus>	 maybe we can close it and leave a TODO for labsdb in 4 year's time
[14:45:08] <marostegui>	 I am fine closing it too
[14:45:15] <jynus>	 once we do dbstore1002
[14:45:23] <marostegui>	 yeah, let's close it with a final comment like that
[14:45:33] <jynus>	 dbstore1002 is blocking too many things
[14:46:07] <marostegui>	 I am going to move T205628 T205627 and T205626 to next
[14:46:08] <stashbot>	 T205628: Handle object metadata backups and compare it with stored database object inventory - https://phabricator.wikimedia.org/T205628
[14:46:10] <stashbot>	 T205626: Document clearly the mariadb backup and recovery setup - https://phabricator.wikimedia.org/T205626
[14:46:11] <stashbot>	 T205627: Purge old metadata for the mariadb backups database - https://phabricator.wikimedia.org/T205627
[14:46:52] <jynus>	 next?
[14:47:02] <marostegui>	 either next or backlog
[14:47:04] <jynus>	 I really don't plan to work on those soon
[14:47:07] <marostegui>	 ah ok!
[14:47:10] <marostegui>	 backlog it is then
[14:47:31] <jynus>	 it is not that we shouldn't but they are low priority
[14:47:43] <jynus>	 e.g. purging will only become an issue in a year or os
[14:47:56] <marostegui>	 T205626 -> this would be nice to have it "soon"
[14:48:10] <jynus>	 and documenting we should do it, but many things are done on the document 
[14:48:27] <jynus>	 and many thing will change from now to implementation
[14:51:01] <wikibugs>	 10DBA, 10Epic, 10User-Elukey: Meta ticket: Migrate multi-source database hosts to multi-instance - https://phabricator.wikimedia.org/T159423 (10Marostegui) 05Open>03Resolved This ticket was meant to be a discussion ticket but turned into a tracking ticket. We are going to close it knowing that what is pe...
[14:51:02] <jynus>	 note the basics are documented, those werre more for in-depth documentation like man pages and api explanations
[14:52:32] <wikibugs>	 10DBA: Document clearly the mariadb backup and recovery setup - https://phabricator.wikimedia.org/T205626 (10Marostegui)
[15:05:04] <marostegui>	 Another transfer.py happy client!
[15:17:03] <wikibugs>	 10DBA: Document clearly the mariadb backup and recovery setup - https://phabricator.wikimedia.org/T205626 (10jcrespo) I've written https://wikitech.wikimedia.org/wiki/MariaDB/Backups#Monitoring_and_metadata_gathering by copying documentation I had written somewhere else.
[15:17:05] <jynus>	 https://wikitech.wikimedia.org/wiki/MariaDB/Backups#Monitoring_and_metadata_gathering
[15:19:11] <marostegui>	 ah cool!!
[16:16:57] <marostegui>	 banyek: will you resume replication on dbstore2002 today? (as you mentioned it today, not sure if that is still the plan)
[16:17:08] <banyek>	 yes
[16:17:39] <marostegui>	 roger
[16:17:47] <banyek>	 that was my plan to do as the very last thing before weekend drops the bass
[16:18:20] <marostegui>	 sure!
[16:18:33] <marostegui>	 just asking :)
[16:18:57] <banyek>	 which is happening now, as the last toku conversion on dbstore1002 is finished as well
[16:18:58] <banyek>	 :)
[16:19:28] <marostegui>	 great
[16:21:20] <wikibugs>	 10DBA, 10User-Banyek: dbstore2002 tables compression status check - https://phabricator.wikimedia.org/T204930 (10Banyek) The compression is stopped for the weekend, and the replication got restarted
[16:22:00] <banyek>	 have a good weekend
[16:22:39] <marostegui>	 you too
[16:23:10] <jynus>	 same
[16:23:14] <marostegui>	 I am off too
[16:23:16] <marostegui>	 o/
[16:23:25] <jynus>	 bye
[17:59:10] <wikibugs>	 10DBA, 10Data-Services, 10Datasets-General-or-Unknown: Archive and drop education program (ep_*) tables on all wikis - https://phabricator.wikimedia.org/T174802 (10Reedy) >>! In T174802#4624526, @Marostegui wrote: > What is pending here? >  > 1) Disable the extension everywhere?  That's done   >>! In T174802...