[05:22:54] <wikibugs>	 10DBA, 06Operations, 10Phabricator, 10ops-eqiad, 13Patch-For-Review: db1048 BBU Faulty - slave lagging - https://phabricator.wikimedia.org/T160731#3306721 (10Marostegui) The battery looks good now, it recharged, the temperature is ok and I have disabled the auto learn. I have started MySQL and once it ca...
[05:59:28] <wikibugs>	 07Blocked-on-schema-change, 10DBA, 13Patch-For-Review: Unify revision table on s3 - https://phabricator.wikimedia.org/T166278#3306742 (10Marostegui)
[06:18:51] <wikibugs>	 07Blocked-on-schema-change, 10DBA, 13Patch-For-Review: Unify revision table on s3 - https://phabricator.wikimedia.org/T166278#3306764 (10Marostegui)
[06:30:08] <wikibugs>	 10DBA, 06Operations, 10Traffic: Substantive HTTP and mediawiki/database traffic coming from a single ip - https://phabricator.wikimedia.org/T166695#3304453 (10Marostegui) That is very strange, the last accesses by the top IP we saw yesterday end up at 19:45 for enwiki. The last access that IP has was at "201...
[07:39:14] <wikibugs>	 10DBA: DROP OAI-related tables - https://phabricator.wikimedia.org/T139342#3306781 (10Marostegui) In order to get ready for the deletion next week I have backuped all the tables across the shards and placed them at: ``` dbstore1001:/srv/tmp/T139342 ```
[09:21:54] <wikibugs>	 10DBA, 10Community-Wikimetrics, 10Icinga, 06Labs, and 2 others: Evaluate future of wmf puppet module "mysql" - https://phabricator.wikimedia.org/T165625#3306961 (10jcrespo) Some modules maybe should install wmf-mariadb101-client ?
[13:14:09] <wikibugs>	 07Blocked-on-schema-change, 10DBA, 13Patch-For-Review: Unify revision table on s3 - https://phabricator.wikimedia.org/T166278#3307370 (10Marostegui)
[14:53:58] <wikibugs>	 10DBA, 06Operations, 10ops-codfw: db2044 cannot install jessie - requires BIOS firmware upgrade - https://phabricator.wikimedia.org/T166683#3307767 (10Papaul) p:05Triage>03Normal
[15:19:31] <wikibugs>	 10DBA, 06Operations, 10ops-codfw: db2044 cannot install jessie - requires BIOS firmware upgrade - https://phabricator.wikimedia.org/T166683#3307874 (10jcrespo) According to my records, 43-70 were bought together. 33-42 are the same exact model, I will check if I can reimage one of those, too.
[15:25:06] <wikibugs>	 10DBA, 06Operations, 10ops-codfw: db2044 cannot install jessie - requires BIOS firmware upgrade - https://phabricator.wikimedia.org/T166683#3307886 (10Papaul) a:05Papaul>03jcrespo Firmware update complete
[15:39:49] <wikibugs>	 10DBA, 06Labs, 10Labs-Infrastructure: Decide whether back-compat views for upcoming major schema changes will be provided in the Labs replicas - https://phabricator.wikimedia.org/T166798#3307956 (10Anomie)
[15:39:58] <wikibugs>	 10DBA, 06Labs, 10Labs-Infrastructure: Decide whether back-compat views for upcoming major schema changes will be provided in the Labs replicas - https://phabricator.wikimedia.org/T166798#3307956 (10Anomie) a:05Anomie>03None
[16:19:19] <wikibugs>	 10DBA, 06Operations, 10ops-codfw: db2044 cannot install jessie - requires BIOS firmware upgrade - https://phabricator.wikimedia.org/T166683#3308073 (10jcrespo) Thank you, taking over for retrying reimage.
[16:57:49] <jynus>	 10.1.24 got released. The changes are not too important, but they contain some of the fixes of bugs I reported
[16:57:57] <jynus>	 I will package it
[18:03:44] <elukey>	 switched to this chan that is less busy
[18:04:14] <elukey>	 jynus: shall I disable huge pages manually (and then patch puppet to make it permanent) and then try to start mysql ?
[18:04:42] <jynus>	 sorry, do not have much time for that, I am busy with other things
[18:05:03] <elukey>	 sure
[18:23:07] <jynus>	 so what's the status, elukey?
[18:24:49] <elukey>	 all seems up at the moment, replication is fine and el_sync works
[18:25:08] <jynus>	 did it recover on start?
[18:25:12] <jynus>	 or not at the end?
[18:25:25] <jynus>	 did it stop cleanly?
[18:26:06] <elukey>	 it stopped cleanly before the shutdown, I only had to turn off huge pages manually
[18:26:10] <elukey>	 and then start mysql
[18:26:20] <jynus>	 no
[18:26:23] <jynus>	 I meant the alter
[18:26:30] <jynus>	 wasn't an alter running on stop?
[18:27:10] <jynus>	 check tomorrow with manuel, I am not up to date with what was going on there
[18:27:22] <jynus>	 maybe some work has to be redone or something
[18:30:07] <elukey>	 sorry I was on the phone
[18:30:40] <elukey>	 I didn't check (my bad) if there was anything ongoing, so I am not sure what was launched.. I'll verify with Manuel (sending an email now)
[18:33:09] <jynus>	 don't worry, shutting down a database server and not check with the dbas is a common mistake
[18:37:39] <jynus>	 probably should add the code for transparent hugepages to puppet
[18:38:33] <elukey>	 I'll surely remember next time, won't do it again :)
[18:41:02] <jynus>	 I actually was serious
[18:41:07] <jynus>	 people tend to do that :-/
[18:41:21] <jynus>	 elukey, https://gerrit.wikimedia.org/r/356648
[18:41:55] <jynus>	 as far as I can see this worked nicely
[18:42:34] <jynus>	 if it had gotten corrupted, it may have taken a month to recover
[18:44:45] <jynus>	 I've changed read only to 0
[18:44:53] <jynus>	 people complain when it is 1
[19:22:04] <wikibugs>	 10DBA, 10Wikimedia-Site-requests, 07Documentation, 05Mediawiki SWAT Deployments: Create summary templates to stop to write the same things everywhere everytime - https://phabricator.wikimedia.org/T165756#3308929 (10Quiddity) 1) Note: Dereckson did some of this in: https://wikitech.wikimedia.org/wiki/Templa...