[00:52:31] 10DBA, 10Beta-Cluster-Infrastructure, 06Operations, 06Release-Engineering-Team: Better mysql command prompt info - https://phabricator.wikimedia.org/T157714#3015480 (10Reedy) [07:04:05] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: db2060 not accessible - https://phabricator.wikimedia.org/T156161#3015955 (10Marostegui) Thanks @Papaul for handling all this. I will get the server ready for you on Monday again [08:30:00] 07Blocked-on-schema-change, 06Collaboration-Team-Triage, 10Notifications, 13Patch-For-Review, 07Schema-change: Add primary key to echo_notification table - https://phabricator.wikimedia.org/T136428#3016016 (10Marostegui) I am going to start altering S3 for the officewiki and mediawiki databases. [08:55:04] 10DBA, 10Beta-Cluster-Infrastructure, 06Operations, 06Release-Engineering-Team, 13Patch-For-Review: Better mysql command prompt info - https://phabricator.wikimedia.org/T157714#3016084 (10jcrespo) p:05Triage>03Low Let's get the ok from beta owners on any changes. [09:01:55] 10DBA, 10Beta-Cluster-Infrastructure, 06Operations, 06Release-Engineering-Team, 13Patch-For-Review: Better mysql command prompt info - https://phabricator.wikimedia.org/T157714#3014147 (10hashar) antoine-approve [09:02:13] jynus: marostegui : the sql prompt on beta, yes definitely :-} [09:51:56] 07Blocked-on-schema-change, 06Collaboration-Team-Triage, 10Notifications, 13Patch-For-Review, 07Schema-change: Add primary key to echo_notification table - https://phabricator.wikimedia.org/T136428#3016311 (10Marostegui) dbstore2001 ``` root@neodymium:~# for i in mediawikiwiki officewiki ; do echo $i; m... [11:06:27] 10DBA, 10Beta-Cluster-Infrastructure, 06Operations, 06Release-Engineering-Team, 13Patch-For-Review: Better mysql command prompt info - https://phabricator.wikimedia.org/T157714#3016508 (10jcrespo) I have merged the above patch as requested, but as I commented there, I do not think that will solve the tic... [11:06:40] 10DBA, 10Beta-Cluster-Infrastructure, 06Operations, 06Release-Engineering-Team, 13Patch-For-Review: Better mysql command prompt info for Beta - https://phabricator.wikimedia.org/T157714#3016509 (10jcrespo) [11:11:20] 10DBA, 10Beta-Cluster-Infrastructure, 06Operations, 06Release-Engineering-Team, 13Patch-For-Review: Better mysql command prompt info for Beta - https://phabricator.wikimedia.org/T157714#3014147 (10Marostegui) We can always add it to the `[client]` section too [11:12:47] 10DBA, 10Beta-Cluster-Infrastructure, 06Operations, 06Release-Engineering-Team, 13Patch-For-Review: Better mysql command prompt info for Beta - https://phabricator.wikimedia.org/T157714#3016526 (10jcrespo) >>! In T157714#3016090, @hashar wrote: > antoine-approve photo You are aging well, you look younge... [11:15:28] 10DBA, 10Beta-Cluster-Infrastructure, 06Operations, 06Release-Engineering-Team, 13Patch-For-Review: Better mysql command prompt info for Beta - https://phabricator.wikimedia.org/T157714#3016543 (10jcrespo) >>! In T157714#3016524, @Marostegui wrote: > We can always add it to the `[client]` section too No... [11:18:25] 10DBA, 10Beta-Cluster-Infrastructure, 06Operations, 06Release-Engineering-Team, 13Patch-For-Review: Better mysql command prompt info for Beta - https://phabricator.wikimedia.org/T157714#3016547 (10hashar) > You are aging well, you look younger now than on the profile photo. I am actually younger on the... [11:27:08] 10DBA, 10Wikimedia-Site-requests, 13Patch-For-Review: Recreate a wiki for Wikimedia Portugal - https://phabricator.wikimedia.org/T126832#3016561 (10jcrespo) This worries me: T157636#3012112 @Dereckson: let's schedule the deletion and the recreation at the same time to avoid issues like this one. [11:36:53] 10DBA, 10Beta-Cluster-Infrastructure, 06Operations, 06Release-Engineering-Team, 13Patch-For-Review: Better mysql command prompt info for Beta - https://phabricator.wikimedia.org/T157714#3016567 (10jcrespo) Have a look on how I handle it on production with an alias- enforce the host staticly rathen than \... [14:13:27] 10DBA, 06Operations, 13Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#3016904 (10jcrespo) @Marostegui Let's assume there is not blockers (which we do have) and make a full replacement plan. [14:16:13] 10DBA, 06Operations, 13Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#3016922 (10jcrespo) [14:16:16] 10DBA, 06Operations, 10ops-eqiad, 13Patch-For-Review: Remove db1002-db1007 from production - https://phabricator.wikimedia.org/T105768#3016921 (10jcrespo) [14:52:16] 10DBA, 10Beta-Cluster-Infrastructure, 06Operations, 06Release-Engineering-Team, 13Patch-For-Review: Better mysql command prompt info for Beta - https://phabricator.wikimedia.org/T157714#3017025 (10hashar) Worth noting, I think most of us use the deployment server deployment-tin.eqiad.wmnet to connect to... [14:55:03] 10DBA, 10Beta-Cluster-Infrastructure, 06Operations, 06Release-Engineering-Team, 13Patch-For-Review: Better mysql command prompt info for Beta - https://phabricator.wikimedia.org/T157714#3017032 (10jcrespo) Whatever you decide, I would be happy to help- this is is more of a client issue rather than server... [14:58:41] jynus: marostegui: I am switch CI to run rake for operations/puppet/mariadb thanks :} [14:59:33] 10DBA, 06Operations, 13Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#3017069 (10Marostegui) >>! In T134476#3016904, @jcrespo wrote: > @Marostegui Let's assume there is not blockers (which we do have) and make a full replacement plan. Sounds good to me [15:02:14] hashar, but things are not on production yet [15:02:25] can I rebase op/puppet ? [15:02:44] that is the change that worries me [15:03:34] are we sure to deploy that on a friday? :) [15:04:03] jynus: the change in ops puppet can wait next week [15:04:15] no need to rush on a friday and risk exploding dozens of databases [15:04:23] it is not worth it [15:04:38] well [15:04:44] actually, I was going to do it now [15:04:52] because it is when I finnaly got the time [15:05:04] I just need no blocker from your side [15:05:14] (I will handle ops side/deployment/etc.) [15:05:16] 07Blocked-on-schema-change, 06Collaboration-Team-Triage, 10Notifications, 13Patch-For-Review, 07Schema-change: Add primary key to echo_notification table - https://phabricator.wikimedia.org/T136428#3017103 (10Marostegui) These are the host in eqiad with the ALTER already done too: dbstore1001 ``` root... [15:08:53] jynus: no blocker form my side :) [15:09:15] thank you [15:09:23] I think I may have missundestand your CR [15:09:39] I think you only wanted to merge the repo [15:09:46] my problems were to deploy into production [15:09:53] not the actual merge [15:10:04] then if the change is merged that implies you have to deploy it to production just after it is merged isn't it ? [15:10:24] well, or block all future merges [15:10:37] s/merge/deploys/ [15:24:38] hashar, please understand that I have to mark linting as low prio for me [15:24:49] not because I think it is not important, which is [15:24:53] I full agree [15:25:09] but because there is now broken things, and this is more like a new feature [15:25:13] I kept poking just to make sure you were aware of that change and would somehow at some point eventually manage to land it :-} [15:25:20] I normally do [15:25:25] it just takes some time [15:25:29] yeah which is fine to me [15:26:05] there is no point in breaking prod or forcing you to put extra hours to fix up a lint issue [15:26:09] as you said, they can wait [15:26:23] and it is entirely fine to me :} [15:26:26] I just saw you like 2 pings within an hour [15:26:38] and was a bit like, I do not understand the rush? [15:27:01] maybe because I commented on both the puppet.git and mariadb.git changes? [15:27:05] no [15:27:11] you commented twice [15:27:32] Feb 3 4:18 PM [15:27:37] "this is pending jynus" [15:27:47] Feb 3 6:03 PM [15:27:52] "this is the last one" [15:28:01] I thought you were going to ping me every hour [15:28:36] https://gerrit.wikimedia.org/r/331329 [15:28:48] one of them letting me know it is ready is enough [15:28:53] yeah sorry :-(((( [15:29:06] if you want to be extra sure, file a task on phab [15:29:15] Luca and I merged a bunch of related changes last friday [15:29:20] no problem, I was only worried because you look worried [15:29:24] to me [15:29:35] so I guess my second message is to give a final status [15:29:44] sorry if you thought I was pressuring you :-/ [15:30:09] well I am always worried and anxious [15:30:40] well, a friend of mine usually tells me [15:30:50] calm down, you are too stressed [15:30:53] so that probably transpire in my writings. Next time I will take a plane to your neighborhood and we can hang out instead :) [15:32:09] in particular, the replication check is sensitive [15:32:38] because a single comma can generate lots of pages [15:32:46] and it failed in the past [15:33:00] so I just need some free time to apply it slowly [15:33:09] (not that it is difficult or anything) [15:34:58] I am on the same line no worries jynus :} [15:53:19] 07Blocked-on-schema-change, 06Collaboration-Team-Triage, 10Notifications, 13Patch-For-Review, 07Schema-change: Add primary key to echo_notification table - https://phabricator.wikimedia.org/T136428#3017215 (10Marostegui) db1015, db1038, db1035 and db1044 are also done: ``` root@neodymium:~# for i in med... [18:07:44] 10DBA, 13Patch-For-Review: Fix dbstore2001 and dbstore2002 - https://phabricator.wikimedia.org/T130128#3017721 (10jcrespo) I have enabled delayed replicationon dbstore2001- it is not dangerous, as worse case scenario it just stops the replication thread more than it should. In particular, it was badly/not comp... [21:12:00] 10DBA, 07Availability: Look into Maria 10 parallel-replication - https://phabricator.wikimedia.org/T85266#3018287 (10jcrespo) http://jfg-mysql.blogspot.com.es/2017/02/metric-for-tuning-parallel-replication-mysql-5-7.html