[01:17:51] <wikibugs>	 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 06Stewards-and-global-tools (Temporary-UserRights): Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605#2995422 (10TTO) Added #blocked-on-schema-change tag. I think we are satisfied that the database aspect of this change...
[02:35:22] <wikibugs>	 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 06Stewards-and-global-tools (Temporary-UserRights): Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605#2995536 (10TTO)
[07:39:51] <wikibugs>	 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 06Stewards-and-global-tools (Temporary-UserRights): Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605#2995700 (10Marostegui) Hey @TTO!  We will take a look, given that it needs to be done in all the wikis it will take a...
[08:30:49] <wikibugs>	 10DBA, 10Analytics: Json_extract available on analytics-store.eqiad.wmnet - https://phabricator.wikimedia.org/T156681#2995733 (10Marostegui) That function isn't available on 10.0 or 10.1, but only on 10.2 (https://mariadb.com/kb/en/mariadb/json_extract/) which we do not have in production. We are currently sta...
[09:19:28] <wikibugs>	 07Blocked-on-schema-change, 06Collaboration-Team-Triage, 10Notifications, 13Patch-For-Review, 07Schema-change: Add primary key to echo_notification table - https://phabricator.wikimedia.org/T136428#2995823 (10Marostegui) So on monday I will start with the used ones in the following order:  - s7 - s3 - x1...
[09:25:22] <wikibugs>	 10DBA, 10MediaWiki-Special-pages, 10Wikimedia-Site-requests, 13Patch-For-Review, 07Wikimedia-log-errors: "Invalid DB key" errors on various special pages - https://phabricator.wikimedia.org/T155091#2995834 (10TTO)
[09:25:40] <wikibugs>	 10DBA, 10MediaWiki-Special-pages, 10Wikimedia-Site-requests, 13Patch-For-Review, 07Wikimedia-log-errors: "Invalid DB key" errors on various special pages - https://phabricator.wikimedia.org/T155091#2933244 (10TTO)
[09:26:23] <wikibugs>	 10DBA, 10MediaWiki-Special-pages, 10Wikimedia-Site-requests, 13Patch-For-Review, 07Wikimedia-log-errors: "Invalid DB key" errors on various special pages - https://phabricator.wikimedia.org/T155091#2933244 (10TTO) If we can review that maintenance script and get it run, we will be able to get rid of thes...
[09:27:29] <wikibugs>	 10DBA, 10MediaWiki-Special-pages, 10Wikimedia-Site-requests, 13Patch-For-Review, 07Wikimedia-log-errors: "Invalid DB key" errors on various special pages - https://phabricator.wikimedia.org/T155091#2995838 (10TTO)
[09:46:10] <moritzm>	 good morning, FYI I'll upgrade trusty DB servers to the latest trusty kernel either later today or on Monday (ofc no reboots, but so that we boot into the most recent kernel when a host gets rebooted for other maint work)
[09:46:30] <marostegui>	 thanks for the heads up moritzm :)
[09:52:34] <jynus>	 moritzm, I am already doing that
[09:52:46] <jynus>	 maybe there is no need by monday
[09:54:51] <moritzm>	 jynus: thanks, all the better :-)
[09:55:19] <moritzm>	 just a headsup they they released a new trusty kernel an hour ago
[09:55:26] <jynus>	 oh
[09:55:43] <jynus>	 so all my reboots are worthless?
[09:57:30] <marostegui>	 any objection to upgrade db1064 (vslow from s4) to 10.0.29? (it is running 10.0.23)
[09:57:52] <jynus>	 not for me
[09:58:07] <marostegui>	 ok, it is a jessie, so I will go ahead and upgrade its mariadb
[10:05:59] <jynus>	 "Delayed replication was introduced in MariaDB 10.2.3"
[10:07:00] <marostegui>	 niiiiice!
[10:07:08] <jynus>	 :-/
[10:07:08] <marostegui>	 better later than never
[10:07:10] <marostegui>	 XD
[10:07:28] <marostegui>	 you thought it was going to be on 10.1?
[10:07:28] <jynus>	 not in a version we do not know we can recover from
[10:07:40] <marostegui>	 ah, yeah, that is another story..
[10:08:45] <jynus>	 maybe it can be backported
[10:09:27] <moritzm>	 not worthless at all, it's just that kernel upgrades are a Sisyphos work anyway since new kernels come 1-2 times a month
[10:11:25] <jynus>	 https://mariadb.atlassian.net/browse/MDEV-7145 this is still open
[10:11:34] <jynus>	 maybe 10.2.3 is unreleased
[10:16:03] <marostegui>	 they didn't even replied :(
[10:20:25] <jynus>	 I searched and there is not such a code on github
[10:20:52] <marostegui>	 for 10.2?
[10:21:16] <jynus>	 all branches
[10:21:23] <marostegui>	 :(
[10:21:33] <marostegui>	 so they are hoping to have it in 10.2
[10:21:37] <marostegui>	 :(
[10:21:48] <wikibugs>	 10DBA, 06Labs, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#2996017 (10Marostegui) db1064 the future master of db1095 for s4 has now been:  - Changed its binary log to ROW base - Upgraded from 10.0.23 to 10.0.29 (an...
[10:27:38] <jynus>	 manuel- the current ticket is: https://jira.mariadb.org/browse/MDEV-7145
[10:28:28] <marostegui>	 ah right
[10:28:36] <marostegui>	 https://jira.mariadb.org/secure/ViewProfile.jspa?name=knielsen added a comment - 2016-10-16 22:02
[10:28:39] <marostegui>	 Pushed into 10.2.3
[10:28:41] <marostegui>	 		
[10:28:53] <marostegui>	 I have subscribed myself to that ticket
[11:07:44] <wikibugs>	 10DBA, 10Social-Tools, 10SocialProfile, 13Patch-For-Review: Running update.php with an SQLite-runned wiki while SocialProfile is installed will return a database query error - https://phabricator.wikimedia.org/T157082#2994951 (10Marostegui) I don't think this is related to #dba, so I am going to remove the...
[11:29:10] <jynus>	 echo -n "$HOSTNAME: " && screen -r | wc -l -> neodymium: 16
[11:30:03] <marostegui>	 i will clean up mines in a bit
[11:30:19] <wikibugs>	 10DBA, 06Labs, 10Labs-Infrastructure: Design a method for keeping user-created tables in sync across labsDBs - https://phabricator.wikimedia.org/T156869#2996136 (10Marostegui) Hi,   By no means what I am going to say is the way to go, but just expressing my opinion and showing a method we are doing in produc...
[11:38:43] <marostegui>	 neodymium: 11 :-)
[12:05:28] <wikibugs>	 10DBA, 06Labs, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#2996223 (10Marostegui) db1095 has finished transferring its snapshot to: es1017:/srv/tmp I have tested to decompress it and it worked fine.   db1095 is bac...
[12:11:13] <jynus>	 maybe 39 can be upgraded while depooled? it is a very old version
[12:11:28] <jynus>	 (it is a suggestion, not a request)
[12:12:03] <marostegui>	 I can reimage it once it's finished the alter
[12:12:14] <jynus>	 no necesarely reimage
[12:12:30] <marostegui>	 wow 10.0.15!
[12:12:31] <jynus>	 I would be happy with just an mariadb upgrade
[12:12:34] <jynus>	 yes
[12:12:40] <jynus>	 that is why I said it
[12:12:46] <marostegui>	 Ubuntu 14.04.1 LTS
[12:13:05] <jynus>	 or mariadb + packages
[12:13:23] <marostegui>	 yeah, let's do mariadb+packages
[12:13:53] <jynus>	 it is not doing any rewrite
[12:14:01] <jynus>	 so not reason to keep it on an old package
[12:14:04] <marostegui>	 how did you check that?
[12:14:27] <jynus>	 the old patch read a .rewrite file on datadir
[12:14:35] <jynus>	 or something along those lines
[12:14:46] <marostegui>	 ah :)
[12:14:55] <jynus>	 (which was why I was worried for enwiki apis)
[12:15:18] <marostegui>	 yeah, but db1072 (or 73, can't remember which one exactly) was doing fine without the rewrite, right?
[12:15:25] <marostegui>	 actually better, no?
[12:15:36] <jynus>	 in the end exactly the same
[12:16:07] <jynus>	 and no major issue since an all upgrade
[12:16:14] <jynus>	 but it may depend on the shard
[12:16:20] <jynus>	 again, not the case here
[16:19:21] <wikibugs>	 10DBA, 10MediaWiki-extensions-ORES, 10Reading Epics, 10Revision-Scoring-As-A-Service-Backlog, 07Performance: rcshow=oresreview is slow - https://phabricator.wikimedia.org/T152585#2996727 (10Fjalapeno)
[16:31:57] <wikibugs>	 10DBA, 06Labs, 10Tool-Labs: u3532__ (=marcmiquel) table using 64G on labsdb1001 - https://phabricator.wikimedia.org/T133322#2228158 (10Huji) @marcmiquel are you done with the clean up? If yes, would you mind closing this task?
[16:42:22] <wikibugs>	 10DBA, 06Operations, 13Patch-For-Review: Set up TLS for MariaDB replication - https://phabricator.wikimedia.org/T111654#2222359 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by jynus on neodymium.eqiad.wmnet for hosts: ``` ['db2039.codfw.wmnet'] ``` The log can be found in `/var/log/wmf-auto-re...
[16:51:51] <wikibugs>	 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: Change rack for servers in s1 in codfw - https://phabricator.wikimedia.org/T156478#2996900 (10jcrespo) 34 is done, I think 62 and 70 are pending.
[17:09:22] <wikibugs>	 10DBA, 06Operations, 13Patch-For-Review: Set up TLS for MariaDB replication - https://phabricator.wikimedia.org/T111654#2996940 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db2039.codfw.wmnet'] ```  and were **ALL** successful.
[18:27:16] <wikibugs>	 10DBA, 06Operations, 13Patch-For-Review: Restart pending mysql hosts with old TLS cert - https://phabricator.wikimedia.org/T152188#2997132 (10jcrespo) Pending hosts: ``` db1036.eqiad.wmnet: cacert db1021.eqiad.wmnet: cacert db1022.eqiad.wmnet: cacert db1015.eqiad.wmnet: cacert db2050.codfw.wmnet: cacert db20...
[20:01:21] <wikibugs>	 10DBA, 06Operations, 10ops-codfw: db2060 not accessible - https://phabricator.wikimedia.org/T156161#2997396 (10Papaul) I will need a maintenance window set for this system on Monday from 10am to 1pm for the controller replacement. Thanks Dear Mr Papaul Tshibamba,  Thank you for contacting Hewlett Packard Ent...