[05:52:12] <wikibugs>	 10DBA, 10MediaWiki-Configuration, 10Operations: Create tool to handle the state of database configuration in MediaWiki in etcd - https://phabricator.wikimedia.org/T197126#4281781 (10Marostegui) This is great! Thanks for getting on with this - these are my first thoughts!  > Depool/pool/warmup >dbconfig [depo...
[05:54:22] <wikibugs>	 10DBA, 10MediaWiki-Configuration, 10Operations: Create tool to handle the state of database configuration in MediaWiki in etcd - https://phabricator.wikimedia.org/T197126#4281782 (10Marostegui) p:05Triage>03Normal
[05:57:47] <wikibugs>	 10DBA, 10Gerrit, 10Operations, 10Phabricator: Massive increase of writes in m3 section - https://phabricator.wikimedia.org/T196840#4281783 (10Marostegui) 05Open>03Resolved a:03mmodell This is now back to normal values:  https://grafana.wikimedia.org/dashboard/db/mysql?panelId=2&fullscreen&orgId=1&var...
[06:55:24] <wikibugs>	 10DBA, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10Patch-For-Review, 10Schema-change: Schema change to make archive.ar_rev_id NOT NULL - https://phabricator.wikimedia.org/T191316#4281840 (10Marostegui)
[06:55:48] <wikibugs>	 10DBA, 10Multi-Content-Revisions, 10Patch-For-Review, 10Schema-change: Schema change to drop archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T192926#4281841 (10Marostegui)
[06:56:10] <wikibugs>	 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Patch-For-Review, 10Wikidata-Ministry-Of-Magic: Schema change for ct_tag_id field to change_tag - https://phabricator.wikimedia.org/T195193#4281842 (10Marostegui)
[07:29:40] <hashar>	 I wrote my first sql in age:   select email_address, count(*) as occurences, group_concat(external_id order by account_id)  from account_external_ids group by email_address having occurences > 1;
[07:29:49] <hashar>	 or really, I love GROUP_CONCAT() :]
[07:49:07] <jynus>	 I improved the slides 200x
[07:51:09] <marostegui>	 That is now a proper presentation!!!
[07:55:29] <volans>	 morning DBAs :)
[07:56:38] <mutante>	 hello DBAs!  
[07:56:40] <volans>	 FYI we should start to roll out debmonitor client across the fleet today. It will be very slow and manually controlled, I'll keep an eye on the m2 cluster graphs too of course.
[07:57:07] <volans>	 but wanted to give you an heads up, also in case you see anything strange on m2 feel free to ping me
[07:58:37] <marostegui>	 cool! thanks for the heads up :)
[08:00:51] <mutante>	 FYI i am about to switch dbtree.wikimedia.org from terbium to mwmaint1001. i got the new grants, confirmed they work, document root is very simple and has the PHP file and config with password. tested with apache-fast-test that i get 200 OK.. just flipping the switch in cache::misc and running puppet is left
[08:01:03] <mutante>	 you dont have any special monitoring on dbtree or something right
[08:01:41] <marostegui>	 Monitoring as something that might page?
[08:02:18] <mutante>	 yea, something that is looking at dbtree and isnt human users
[08:02:35] <marostegui>	 I have never seen a page for dbtree, so I would assume we don't
[08:03:24] <mutante>	 ok, ill do it and switching back and forth is easy
[08:10:13] <mutante>	 merged the cache::misc config, running puppet on misc cp*
[08:10:25] <mutante>	 watching apache logs for dbtree
[08:14:12] <mutante>	 it makes a connection to google.com each time ...
[08:14:57] <mutante>	 it uses google.com/jsapi for visualization-orgchart
[08:16:14] <marostegui>	 https://phabricator.wikimedia.org/T96499
[08:17:05] <mutante>	 ah :) 
[08:19:07] <Hauskatze>	 mutante: is wasat going to change too?
[08:19:27] <mutante>	 need to pupptieze that libapache2-mod-php gets installeds
[08:19:31] <mutante>	 Hauskatze: yes
[08:19:39] * Hauskatze tries to understand where the 1001 comes from
[08:19:44] <mutante>	 i wasnt 100% sure about the order though 
[08:19:58] <mutante>	 1 means eqiad
[08:20:27] <mutante>	 the 1 at the beginning. 1 eqiad, 2 codfw, 3 esams, 4 ulsfo, 5 eqsin
[08:20:35] <mutante>	 and 001 is just where you start counting
[08:20:54] <mutante>	 room for 999 maitnenance servers
[08:20:54] <Hauskatze>	 got it!
[08:21:02] <Hauskatze>	 thanks
[08:21:24] <Hauskatze>	 so wasat is going to be... mwmaint2001?
[08:21:29] <mutante>	 correct
[08:21:33] <Hauskatze>	 chachi
[08:22:36] <wikibugs>	 10DBA, 10MediaWiki-Configuration, 10Operations: Create tool to handle the state of database configuration in MediaWiki in etcd - https://phabricator.wikimedia.org/T197126#4282045 (10jcrespo) > I think that if NAME is just the hostname, it should show the config for all the configured HOST:PORT combinations i...
[08:23:09] <mutante>	 awww.. fail:  dbtree is using mysql_connect
[08:23:19] <mutante>	 and that isnt a thing anymore on stretch with PHP7
[08:23:25] <mutante>	 need to convert to PDO or mysqli
[08:23:37] <mutante>	 PHP Fatal error:  Uncaught Error: Call to undefined function mysql_connect() 
[08:23:56] <mutante>	 i will revert first 
[08:26:14] <wikibugs>	 10DBA, 10MediaWiki-Configuration, 10Operations: Create tool to handle the state of database configuration in MediaWiki in etcd - https://phabricator.wikimedia.org/T197126#4282069 (10Joe) >>! In T197126#4280521, @Volans wrote: > Quick first feedback/questions on the proposal: >  >> dbconfig get NAME gets you...
[08:26:18] <wikibugs>	 10DBA, 10MediaWiki-Configuration, 10Operations: Create tool to handle the state of database configuration in MediaWiki in etcd - https://phabricator.wikimedia.org/T197126#4282070 (10Joe)
[08:28:39] <wikibugs>	 10DBA, 10MediaWiki-Configuration, 10Operations: Create tool to handle the state of database configuration in MediaWiki in etcd - https://phabricator.wikimedia.org/T197126#4282078 (10Volans) @Joe ack to all your replies, thanks for integrating the suggestions!
[08:31:13] <wikibugs>	 10DBA, 10MediaWiki-Configuration, 10Operations: Create tool to handle the state of database configuration in MediaWiki in etcd - https://phabricator.wikimedia.org/T197126#4282085 (10Joe) >>! In T197126#4282045, @jcrespo wrote: > After thinking for a while, `pool|depool|warmup` (as an interface, not as the id...
[08:32:18] <wikibugs>	 10DBA, 10MediaWiki-Configuration, 10Operations: Create tool to handle the state of database configuration in MediaWiki in etcd - https://phabricator.wikimedia.org/T197126#4282100 (10Joe)
[08:35:51] <wikibugs>	 10DBA, 10MediaWiki-Configuration, 10Operations: Create tool to handle the state of database configuration in MediaWiki in etcd - https://phabricator.wikimedia.org/T197126#4282109 (10Joe) >>! In T197126#4281781, @Marostegui wrote: > This is great! Thanks for getting on with this - these are my first thoughts!...
[08:36:56] <wikibugs>	 10DBA, 10MediaWiki-Configuration, 10Operations: Create tool to handle the state of database configuration in MediaWiki in etcd - https://phabricator.wikimedia.org/T197126#4282110 (10Joe)
[08:40:14] <_joe_>	 thanks for the input, jynus marostegui 
[08:40:17] <_joe_>	 very useful
[08:40:33] <marostegui>	 no, thank you for getting on with this, it will be a massive win
[08:41:28] <volans>	 marostegui: as you recently informally tagged some hosts as 'candidate emergency master', would it be useful to have that info in etcd too?
[08:41:46] <volans>	 I know it's not mw-specific, but kinda related
[08:42:07] <marostegui>	 volans: indeed, that is a good point
[08:42:32] <wikibugs>	 10DBA, 10Operations, 10Traffic, 10WMF-Legal, and 3 others: dbtree loads third party resources (from jquery.com and google.com) - https://phabricator.wikimedia.org/T96499#4282120 (10Bawolff)
[08:44:57] <_joe_>	 adding new fields to a schema is easy
[08:45:01] <_joe_>	 removing them is hard
[08:45:07] <_joe_>	 as usual :D
[08:49:43] <jynus>	 _joe_ volans, please let's start with a much smaller scope
[08:49:44] <jynus>	 it is ok to plan for the future
[08:50:15] <_joe_>	 what is a smaller scope here?
[08:50:25] <_joe_>	 I mean I'm happy to implement an MVP
[08:50:36] <_joe_>	 but I need you and marostegui to tell me what you consider that
[08:51:04] <wikibugs>	 10DBA, 10MediaWiki-Configuration, 10Operations: Create tool to handle the state of database configuration in MediaWiki in etcd - https://phabricator.wikimedia.org/T197126#4282151 (10Marostegui) >>! In T197126#4282109, @Joe wrote: >>  >> This is not likely to happen in a near future, but as we are starting fr...
[08:51:05] <jynus>	 sorry, got disconnected
[08:51:10] <jynus>	 checking the public logs
[08:51:12] <marostegui>	 _joe_: For me, a simply: pool/depool slave
[08:51:17] <marostegui>	 "simply"
[08:51:48] <_joe_>	 marostegui: well most of the things will need to be done anyways, esp in terms of the data model
[08:52:03] <marostegui>	 yeah, that is the hardest part I would say
[08:52:13] <jynus>	 I was explaining myself: "but with sanity checks and other stuff, let's not overthink right now, because things may change"
[08:52:25] <jynus>	 "I hope you understand my fears and my willigness to have very very small sprints- like, make read only by section work first"
[08:52:59] <_joe_>	 well the point of writing sanity checks now is to create the structure in the code to support them, mostly
[08:53:08] <jynus>	 sure
[08:53:20] <_joe_>	 anyways, I'll see what I can come up with
[08:53:25] <jynus>	 just that is the part that will most likely change
[08:53:36] <jynus>	 because as we use it, we will make mistakes
[08:53:42] <jynus>	 and know what to check :-)
[08:54:09] <jynus>	 (or know which ones prevent us from doing things we want to do)
[09:20:12] <jynus>	 I've added https://wikitech.wikimedia.org/wiki/MariaDB#Decommissioning_a_mariadb_host_server_checklist
[09:20:20] <jynus>	 feel free to add more things to it
[09:20:53] <marostegui>	 ah nice
[09:21:03] <marostegui>	 I will add something
[09:21:09] <jynus>	 I allways forget something
[09:21:39] <jynus>	 we could even substitute generic steps on decom checklist (any op part) with that
[09:22:11] <jynus>	 I will add some details on some steps later
[10:24:54] <mark>	 jynus: marostegui: shall we discuss (and finalize) goals next week?
[10:25:05] <marostegui>	 mark: we are at the offiste :)
[10:25:17] <mark>	 ?
[10:25:32] <marostegui>	 mark: next week during the offsite you mean?
[10:25:44] <mark>	 yes :)
[10:25:50] <marostegui>	 then yes :)
[10:25:53] <mark>	 hmm need to find a slot before manuel leaves
[10:25:56] <mark>	 what time on wed do you leave?
[10:26:01] <marostegui>	 let me check
[10:26:16] <jynus>	 mark we already kind of decided already
[10:26:37] <marostegui>	 flight leaves at 3:30pm
[10:26:42] <jynus>	 we only need your ok on language and some details if we can have help for some things
[10:26:48] <mark>	 marostegui: so until lunch more or less
[10:26:49] <mark>	 ok
[10:26:54] <marostegui>	 kind of yes
[10:27:34] <mark>	 what is your proposal?
[10:28:14] <marostegui>	 mark: line #193
[10:28:24] <marostegui>	 but there are lots of ideas there, and we need to reduce the scope
[10:28:34] <jynus>	 or get help
[10:28:56] <marostegui>	 from line 193 to 201
[10:29:28] <marostegui>	 that is what we would like to reach at some point, now, we need to decide what is a doable goal to start with 
[10:29:35] <marostegui>	 as doing all that in a Q is impossible
[10:30:19] <mark>	 i think we need a concrete plan first of all
[10:30:27] <mark>	 is there one already? like a ticket?
[10:31:13] <marostegui>	 We kinda have one https://phabricator.wikimedia.org/T156461
[10:34:48] <mark>	 right
[10:34:55] <mark>	 that's a ticket, not much of a plan yet
[10:35:09] <mark>	 but at least it breaks it up into some subgoals
[10:35:12] <marostegui>	 yeah
[10:35:22] <marostegui>	 that is the thing, we kind of know where we want to go to
[10:35:23] <mark>	 which one do you guys feel is now missed the most?
[10:36:38] <marostegui>	 Probably line 185 
[10:37:12] <mark>	 so that's probably https://phabricator.wikimedia.org/T196366
[10:38:04] <marostegui>	 yeah, that is it
[10:38:08] <mark>	 right
[10:38:37] <mark>	 so I suppose the most important thing is to get a description on this ticket of what repl.pl does that does not work when the master is unavailable
[10:38:41] <mark>	 and what should happen instead
[10:39:39] <mark>	 i really don't know the details, but I suspect the problem is that it can't check the state of the master e.g. binlog positions etc
[10:39:47] <marostegui>	 yeah, that is mostly it
[10:39:50] <jynus>	 there is many issues
[10:39:55] <jynus>	 how deep we want to go
[10:40:02] <jynus>	 is what we have not decided yet
[10:40:06] <mark>	 so now you guys should figure out and describe here what it can do and should do instead for the most common cases
[10:40:19] <jynus>	 because some things requires, for example, setup a source of truth
[10:40:19] <marostegui>	 If you check like #188, those are dependencies for line 185 and 186
[10:40:21] <mark>	 and either the goal needs to be or at least include that
[10:42:04] <jynus>	 or depend on mediawiki/etc/cumin changes
[10:43:00] <mark>	 so this needs a lot of discussion still
[10:43:05] <mark>	 i propose we have a session on it at the offsite
[10:43:14] <marostegui>	 sounds good to me
[10:43:19] <mark>	 too bad joe won't make it
[10:43:20] <mark>	 that doesn't help
[10:43:21] <volans>	 +1
[10:43:25] <mark>	 but riccardo can join
[10:43:25] <jynus>	 the idea is clear, the depth an dependencies and scope is not
[10:43:28] <mark>	 yeah
[10:43:34] <mark>	 and i'm not sure we can finish that in a week
[10:43:45] <mark>	 we'll see
[10:43:58] <mark>	 it's a very useful and worthwhile goal
[10:44:44] <jynus>	 the alternative was backups, but we wanted to wait to have the hardware purchased on Q1 and set it up and have a goal on Q2
[10:44:45] <mark>	 ...and we should probably replace repl with a modern python script along the way :P
[10:44:55] <jynus>	 yes, that is the plan
[10:45:02] <mark>	 the hardware has already been ordered to some extent
[10:45:07] <jynus>	 ?
[10:45:17] <mark>	 partially
[10:45:23] <jynus>	 db specific hardware
[10:45:27] <mark>	 yeah not that
[10:45:35] <mark>	 we may need to do two goals
[10:45:37] <mark>	 two small ones
[10:45:41] <mark>	 so a bit on backups, and some on this
[10:46:01] <jynus>	 this, on its minimum expresion is doable
[10:46:16] <jynus>	 "writing 2 scripts"
[10:46:28] <mark>	 and yeah
[10:46:37] <mark>	 if we do this automation goal I want to make sure we have riccardo involved
[10:46:43] <mark>	 strongly involved
[10:46:49] <jynus>	 that was our main question
[10:46:50] <mark>	 and we're hiring him help ;)
[10:46:54] <jynus>	 will he have him?
[10:47:01] <mark>	 i have no idea what riccardo/faidon are planning for next quarter
[10:47:03] <mark>	 will find out! :)
[10:47:07] <jynus>	 if not, we will reduce it a lot
[10:47:29] <volans>	 yeah, not decided yet, bunch of ideas as always
[10:47:42] <jynus>	 same for etcd/app server
[10:47:52] <mark>	 yes
[10:47:58] <jynus>	 if they don't have time and have other priorities (understandable)
[10:48:05] <mark>	 we'll see
[10:48:09] <jynus>	 we will delay that
[10:48:17] <marostegui>	 and fallback to backups
[10:48:17] <mark>	 i do consider this pretty important and something we (you!) spend a lot of time on
[10:48:42] <jynus>	 the problem with backups is taht without hardware there is not much to progress :-/
[10:48:46] <mark>	 of course
[10:48:53] <jynus>	 we could do software
[10:48:55] <mark>	 we can start working on ordering that immediately
[10:49:01] <mark>	 it -should- not take long
[10:49:02] <marostegui>	 yeah, we could work on monitoring
[10:49:04] <jynus>	 binary backups
[10:49:04] <marostegui>	 (alerting)
[10:49:05] <mark>	 (but I said that last time too eh)
[10:49:06] <jynus>	 monitoring
[10:49:06] <marostegui>	 stuff like that
[10:49:11] <mark>	 yes
[10:49:16] <jynus>	 binlog backups
[10:49:17] <jynus>	 etc.
[10:49:27] <jynus>	 that has very little dependencies except our time
[10:49:45] <jynus>	 and have it ready for when hardware arrives
[10:50:36] <mark>	 yes
[10:50:45] <mark>	 good thing is
[10:50:53] <mark>	 both automation and backups are part of our annual plans for the next fiscal year
[10:50:55] <mark>	 which starts now
[10:51:13] <mark>	 (even better where they combine ;)
[11:22:31] <marostegui>	 mark: Check line #182, we have developed a bit the backups goal proposal, instead of the automation one which might be for Q2
[11:22:55] <mark>	 i hope we can make some progress on both if possible
[11:23:51] <mark>	 that looks good yes
[11:23:59] <marostegui>	 Yeah, but both as goal might be too much, we can always work behind the scenes
[11:25:29] <mark>	 we can reduce the goals so they're not too much
[11:26:01] <mark>	 for example, for the db replica change automation it sounds we have some more discussion and planning to do
[11:26:05] <mark>	 that by itself could be a goal perhaps
[11:26:23] <marostegui>	 like a plan?
[11:26:32] <marostegui>	 or some implementation details?
[11:26:40] <mark>	 yeah, to have a concrete plan so it can easily be implemented in Q2
[11:27:03] <mark>	 with /some/ implementation details, as I described above
[11:27:15] <mark>	 so right now the ticket says that repl.pl doesn't work but it doesn't explain why
[11:27:30] <mark>	 so we need to figure out why it doesn't work and how we want the replacement to work instead
[11:27:44] <mark>	 that's largely DBA specific knowledge, too
[11:30:12] <marostegui>	 yeah, I get it (I think)
[11:30:16] <mark>	 but hey
[11:30:19] <marostegui>	 So let's discuss that part in the offsite
[11:30:22] <mark>	 it's entirely possible we all sit in a room next week
[11:30:26] <mark>	 and come out with a proposal that works
[11:30:29] <mark>	 before the goal even starts
[11:30:31] <mark>	 who knows!
[11:30:37] <marostegui>	 so the backups one looks good?
[11:30:40] <mark>	 and then riccardo implements it next week ;p
[11:30:50] <mark>	 backups proposal mostly looks good
[11:30:54] <mark>	 language needs some tweaks still
[11:30:56] <marostegui>	 yeah
[11:30:56] <mark>	 but i can help with that ;)
[11:31:04] <marostegui>	 so, let's discuss the other one next week
[11:31:10] * volans hides
[11:33:42] <mark>	 made a few tweaks already, what do you think?
[11:34:09] <marostegui>	 looks good!
[11:41:07] <mark>	 so how would we detect 'incorrect generation'?
[11:41:49] <jynus>	 that is part of the goal, generate stats and detect anomaly states
[11:42:11] <jynus>	 which ones are to be seen, but from the top of my mind: 0 size files
[11:42:22] <jynus>	 tables with less than 80% of the previous size
[11:42:31] <jynus>	 tables with over 20% of the previous size
[11:42:38] <jynus>	 objects disapearing
[11:42:55] <jynus>	 (new databases, new tables, no databases, no tables)
[11:43:16] <jynus>	 based on lists we have of things that are expected and previous week stats
[11:43:34] <jynus>	 we were discussing if to do stats on prometheus or on tendril
[11:44:15] <jynus>	 that is aside from the obvious thing- backups script returns non-0 state and parsing of log files :-)
[11:44:54] <jynus>	 the advance monitoring would be outside and be part of backup testing
[11:45:01] <jynus>	 *not part of the scope
[11:45:18] <jynus>	 -recovering to the test hosts and let it replicate/run compare.py on them
[11:45:36] <mark>	 right
[11:45:38] <jynus>	 that is backup testing and it is a separate goal on itslef
[11:45:50] <mark>	 yes agreed
[12:58:00] <wikibugs>	 10DBA, 10Data-Services, 10cloud-services-team: Maintain-views and maintain_meta-p scripts shouldn't run if mysql-upgrade is running - https://phabricator.wikimedia.org/T184540#4282705 (10Marostegui) Lately the way @Bstorm and myself have been working with this is basically depooling the hosts.  She'd send th...
[13:00:19] <wikibugs>	 10DBA, 10Data-Services, 10cloud-services-team: Maintain-views and maintain_meta-p scripts shouldn't run if mysql-upgrade is running - https://phabricator.wikimedia.org/T184540#4282720 (10jcrespo) > There is probably not much stuff we can do about this rather than just coordinating and have good communication...
[13:19:05] <wikibugs>	 10DBA, 10Data-Services, 10cloud-services-team: Maintain-views and maintain_meta-p scripts shouldn't run if mysql-upgrade is running - https://phabricator.wikimedia.org/T184540#4282769 (10Marostegui) 05Open>03Resolved