[05:34:05] <wikibugs_>	 10DBA: Drop flaggedrevs tables on wikis where it is not enabled - https://phabricator.wikimedia.org/T174801#4177490 (10Marostegui)
[05:34:07] <wikibugs_>	 10DBA: Drop flaggedrevs tables at eswiki - https://phabricator.wikimedia.org/T193678#4177486 (10Marostegui) 05Open>03Resolved p:05Triage>03Normal a:03Marostegui They were all empty, so I have dropped them
[05:36:05] <wikibugs_>	 10DBA: Drop flaggedrevs tables at metawiki - https://phabricator.wikimedia.org/T193390#4167712 (10Marostegui) I would assume the data is necessary to be backed up, no? There is only data on these tables: ``` flaggedrevs +----------+ | count(*) | +----------+ |      185 | +----------+ flaggedrevs_promote +-------...
[05:37:26] <wikibugs_>	 10DBA: Drop flaggedrevs tables at eswikibooks - https://phabricator.wikimedia.org/T193676#4177494 (10Marostegui) p:05Triage>03Normal There is not much data, I assume it is not needed to back it up? ``` flaggedimages +----------+ | count(*) | +----------+ |       43 | +----------+ flaggedpage_config +--------...
[05:39:48] <wikibugs_>	 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy - https://phabricator.wikimedia.org/T190704#4177496 (10Marostegui) db1116 is now replicating a multi-instance sanitized copy (also checked with check_private_data) of the...
[05:40:29] <wikibugs_>	 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy - https://phabricator.wikimedia.org/T190704#4177497 (10Marostegui)
[05:47:14] <wikibugs_>	 10DBA: Drop flaggedrevs tables on wikis where it is not enabled - https://phabricator.wikimedia.org/T174801#4177502 (10Marostegui)
[05:55:28] <wikibugs_>	 10DBA: Drop flaggedrevs tables at metawiki - https://phabricator.wikimedia.org/T193390#4177505 (10Marostegui) I have backuped them just in case and left them at: `dbstore1001:/srv/backups/tmp/T193390.tar.gz`
[06:02:23] <wikibugs_>	 10DBA: Drop flaggedrevs tables on wikis where it is not enabled - https://phabricator.wikimedia.org/T174801#4177517 (10Marostegui)
[06:02:25] <wikibugs_>	 10DBA: Drop flaggedrevs tables at metawiki - https://phabricator.wikimedia.org/T193390#4177513 (10Marostegui) 05Open>03Resolved p:05Triage>03Normal a:03Marostegui Dropped them
[06:02:41] <wikibugs_>	 10DBA: Drop flaggedrevs tables on wikis where it is not enabled - https://phabricator.wikimedia.org/T174801#3573416 (10Marostegui)
[07:24:58] <wikibugs_>	 10DBA: Drop flaggedrevs tables at eswikibooks - https://phabricator.wikimedia.org/T193676#4177599 (10Marostegui) Left a backup at: `dbstore1001:/srv/backups/tmp/T193676.tar.gz`
[07:28:11] <wikibugs_>	 10DBA: Drop flaggedrevs tables on wikis where it is not enabled - https://phabricator.wikimedia.org/T174801#4177616 (10Marostegui)
[07:28:13] <wikibugs_>	 10DBA: Drop flaggedrevs tables at eswikibooks - https://phabricator.wikimedia.org/T193676#4177613 (10Marostegui) 05Open>03Resolved a:03Marostegui Tables dropped
[07:28:22] <wikibugs_>	 10DBA: Drop flaggedrevs tables on wikis where it is not enabled - https://phabricator.wikimedia.org/T174801#3573416 (10Marostegui)
[07:34:03] <wikibugs_>	 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: db2081 crashed/rebooted, probably due to hardware failure - https://phabricator.wikimedia.org/T193325#4177621 (10jcrespo) The check detected some difference, but they could be false positives, checking again.
[08:02:30] <wikibugs_>	 10DBA, 10Operations-Software-Development, 10Patch-For-Review: Debmonitor: request for misc DB allocation - https://phabricator.wikimedia.org/T192875#4152620 (10jcrespo) a:03jcrespo
[08:27:22] <volans>	 jynus: for debmonitor, assuming that it will end up on m2, what DNS/IP should I put in the config? (I'm just working on the puppet patches)
[08:27:45] <jynus>	 m2-master.eqiad.wmnet
[08:28:00] <jynus>	 we don't yet have a proper deployment on codfw
[08:28:10] <jynus>	 but it is not yet ready
[08:28:52] <volans>	 ack, thanks
[08:29:36] <jynus>	 I was checking the available collations and I ended up filing a mysql bug
[08:29:44] <volans>	 lol
[08:29:55] <marostegui>	 url!
[08:29:58] <volans>	 as always when getting down a rabbit hole
[08:30:52] <volans>	 but don't worry, I know is not yet there (the DB account) and most CRs are still pending final review, so don't worry
[08:30:52] <jynus>	 so I will create the database as utf8mb4_unicode_520_ci
[08:30:54] <marostegui>	 This is jynus http://i.imgur.com/QMFZ8Zq.gif
[08:30:55] <volans>	 it's not a blocker
[08:31:07] <jynus>	 because there is not yet recent unicode support
[08:34:17] <jynus>	 but that doesn't support emoji
[08:35:59] <volans>	 lol, I don't plan to have emojis in debmonitor :D
[08:36:09] <jynus>	 this is not because of you
[08:36:24] <jynus>	 we need its support, and I am testing it at the same time
[08:36:38] <volans>	 I imagined ;)
[08:41:12] <wikibugs_>	 10DBA, 10Operations-Software-Development, 10Patch-For-Review: Debmonitor: request for misc DB allocation - https://phabricator.wikimedia.org/T192875#4177729 (10jcrespo) ``` root@db1051[debmonitor]> SELECT * FROM test; +------+ | c    | +------+ | 💩     | +------+ 1 row in set (0.00 sec)  root@db1051[debmonit...
[08:42:53] <volans>	 thanks for the encouraging emoji :-P
[08:45:36] <marostegui>	 xddddddd
[08:48:04] <jynus>	 volans: do you really need all privileges from the web interfaces?
[08:48:20] <jynus>	 normally the web interfaces get select, or modify rights at most
[08:48:26] <jynus>	 and all on admin interfaces
[08:48:38] <jynus>	 but I don't know how that that works for you applicatino
[08:49:51] <volans>	 sooo, the web interface surely need select/insert/updates because the data will come through it. The schema changes are made by a CLI (django manage.py migrate) and I still need to fully understand if that will be done by a scap step from the deployment hosts
[08:50:09] <volans>	 in that case we could have a separate account there
[08:50:11] <jynus>	 I will add all
[08:50:15] <jynus>	 no problem
[08:50:19] <jynus>	 just a heads up
[08:52:16] <volans>	 ack, as soon as I fully understand the deploy procedure I'll let you know how we can split it
[08:52:23] <volans>	 for a more safer config
[09:07:28] <wikibugs_>	 10DBA, 10Operations-Software-Development, 10Patch-For-Review: Debmonitor: request for misc DB allocation - https://phabricator.wikimedia.org/T192875#4177782 (10jcrespo) @volans You can test it now, but remember this is a full production host- no load testing or other silly things, as this shares service with...
[09:15:13] <wikibugs_>	 10DBA, 10Operations-Software-Development, 10Patch-For-Review: Debmonitor: request for misc DB allocation - https://phabricator.wikimedia.org/T192875#4177818 (10jcrespo) Backups added on codfw (there are not, at the time, backups for m2 on eqiad) for dbmonitor.
[09:17:27] <wikibugs_>	 10DBA, 10Operations-Software-Development, 10Patch-For-Review: Debmonitor: request for misc DB allocation - https://phabricator.wikimedia.org/T192875#4177831 (10jcrespo) a:05jcrespo>03Volans Assigning to you for verification and read the caveats at T192875#4177782.
[09:24:23] <wikibugs_>	 10DBA, 10Operations-Software-Development, 10Patch-For-Review: Debmonitor: request for misc DB allocation - https://phabricator.wikimedia.org/T192875#4177848 (10Volans) Great! Thanks a lot.  >>! In T192875#4177782, @jcrespo wrote: > @volans You can test it now, but remember this is a full production host- no...
[09:38:51] <wikibugs_>	 10DBA: Drop flaggedrevs tables on wikis where it is not enabled - https://phabricator.wikimedia.org/T174801#4177876 (10MarcoAurelio)
[09:56:03] <Amir1>	 jynus: please ping me when you have a little bit of time
[10:06:04] <mark>	 so
[10:06:05] <mark>	 jynus: marostegui: here?
[10:06:06] <mark>	 hangouts.google.com cannot be resolved
[10:06:21] <marostegui>	 o/
[10:11:44] <mark>	 yeah this is not working
[10:11:44] <mark>	 shall we skip this meeting? :)
[10:11:44] <mark>	 or discuss things here
[10:12:09] <volans>	 I guess his connection is the one not working :)
[10:12:10] <marostegui>	 sure
[11:00:34] <godog>	 smart health alerts downtime expired, which reminded me I should ping you about what to do with those
[11:01:13] <jynus>	 ^ marostegui let'a talk after lunch?
[11:01:30] <marostegui>	 sure
[11:01:31] <godog>	 works for me, I'm going to lunch shortly
[11:01:39] <marostegui>	 IO am going for lunch now
[12:30:41] <marostegui>	 godog jynus my proposal for the alerts: https://phabricator.wikimedia.org/P7073
[12:37:34] <marostegui>	 jynus: are you touching s4 codfw? I would like to deploy a schema change directly on codfw master with replication
[13:01:40] <wikibugs_>	 10DBA, 10Patch-For-Review: Productionize 8 eqiad hosts - https://phabricator.wikimedia.org/T192979#4178179 (10jcrespo)
[13:02:02] <jynus>	 no
[13:02:12] <marostegui>	 I'll proceed then!
[13:07:37] <wikibugs_>	 10DBA: Decommission db1060 - https://phabricator.wikimedia.org/T193732#4178195 (10Marostegui) p:05Triage>03Normal
[13:21:41] <godog>	 marostegui: plan looks good to me! 
[14:03:35] <wikibugs_>	 10DBA: Decommission db1056 - https://phabricator.wikimedia.org/T193736#4178332 (10jcrespo) p:05Triage>03Normal
[14:05:20] <wikibugs_>	 10DBA: Decommission db1056 - https://phabricator.wikimedia.org/T193736#4178383 (10jcrespo) a:03jcrespo
[14:37:13] <godog>	 I'll open tasks for the remaining hosts
[14:37:37] <marostegui>	 godog: No need to I think
[14:37:53] <marostegui>	 I will manually fail them and the automated alert will create the degraded raid one
[14:38:02] <godog>	 oh that's right, even better
[14:38:03] <godog>	 thanks!
[14:39:29] <marostegui>	 I will fail them on Tuesday, don't want to leave them failed during the weekend
[14:39:44] <marostegui>	 Actually, I will check if chris is at the dc now
[14:39:45] <marostegui>	 and fail one
[14:40:33] <volans>	 you're lazy :-P
[14:40:46] <volans>	 get the automation work for you and create tasks for you
[14:40:57] <marostegui>	 Just to check if it keeps working
[14:40:58] <marostegui>	 :p
[14:41:31] <volans>	 sure sure, good excuse though ;)
[14:43:19] <jynus>	 volans: when it works
[14:43:46] <jynus>	 #roast
[14:45:01] <marostegui>	 can someone double check that the wrong disk on db1063 is 1 so: megacli -PDOffline -PhysDrv \[32:1\] -aALL
[14:45:04] <marostegui>	 ?
[14:45:22] <jynus>	 it should say it on the alert
[14:45:30] <marostegui>	 it does :)
[14:45:47] <marostegui>	 but I want another set of eyes if that's possible
[14:46:19] <volans>	 $ sudo /usr/local/lib/nagios/plugins/get-raid-status-megacli
[14:46:37] <volans>	 Enclosure Device ID: 32 ; Slot Number: 1
[14:46:39] <marostegui>	 :)
[14:46:58] <marostegui>	 thanks
[14:47:33] <volans>	 de nada
[14:55:05] <jynus>	 sorry, got distracted by manager
[14:55:45] <wikibugs_>	 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1063 - https://phabricator.wikimedia.org/T193747#4178575 (10Marostegui) p:05Triage>03Normal a:03Cmjohnson This is m1 master - we failed the disk manually as it has errors
[15:05:13] <wikibugs_>	 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1063 - https://phabricator.wikimedia.org/T193747#4178568 (10RobH) The warranty on db1063 has expired, and is no longer under warranty support.  Any failed disks will need to be replaced from shelf spares.
[15:26:39] <cmjohnson1>	 marostegui the disk in slot 1 on db1063 has been replaced
[15:26:47] <marostegui>	 \o/
[15:26:49] <marostegui>	 Thanks cmjohnson1
[15:28:41] <wikibugs_>	 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1063 - https://phabricator.wikimedia.org/T193747#4178684 (10Marostegui) Disk replaced by @Cmjohnson  ``` root@db1063:~# megacli -PDRbld -ShowProg -PhysDrv [32:1] -aALL  Rebuild Progress on Device at Enclosure 32, Slot 1 Completed 2% in 1 Minutes. ```
[15:38:45] <wikibugs_>	 10DBA, 10Operations, 10hardware-requests, 10ops-codfw, 10Patch-For-Review: Decommission db2011 - https://phabricator.wikimedia.org/T187886#4178726 (10Papaul)
[15:39:11] <wikibugs_>	 10DBA, 10Operations, 10hardware-requests, 10ops-codfw, 10Patch-For-Review: Decommission db2011 - https://phabricator.wikimedia.org/T187886#3989185 (10Papaul) a:05Papaul>03RobH @RobH  done on my end
[15:42:28] <wikibugs_>	 10DBA, 10Operations, 10hardware-requests, 10ops-codfw: Decommission db2011 - https://phabricator.wikimedia.org/T187886#4178762 (10RobH)
[15:42:33] <wikibugs_>	 10DBA, 10Operations, 10hardware-requests, 10ops-codfw: Decommission db2011 - https://phabricator.wikimedia.org/T187886#3989185 (10RobH) 05Open>03Resolved
[15:42:36] <wikibugs_>	 10DBA, 10Operations, 10Patch-For-Review: Setup newer machines and replace all old misc (m*) and x1 codfw machines - https://phabricator.wikimedia.org/T183470#4178768 (10RobH)
[15:44:00] <wikibugs_>	 10DBA, 10Goal, 10Patch-For-Review: Decommission database hosts <= db2031 (tracking) - https://phabricator.wikimedia.org/T176243#4178782 (10Marostegui) 05Open>03Resolved This task is now completely done. Thanks everyone who helped on getting this goal finished!
[15:44:02] <wikibugs_>	 10DBA, 10Goal, 10Patch-For-Review: Decommission database hosts <= db2031 (tracking) - https://phabricator.wikimedia.org/T176243#4178784 (10Marostegui) This task is now completely done. Thanks everyone who helped on getting this goal finished!
[16:52:10] <wikibugs_>	 10DBA, 10Operations: Multiple pages with no revisions - https://phabricator.wikimedia.org/T112282#4179032 (10RobH) p:05High>03Low
[22:42:27] <wikibugs_>	 10DBA, 10Cloud-Services, 10Patch-For-Review: Make watchlist table available as curated foo_p.watchlist_count on labsdb - https://phabricator.wikimedia.org/T59617#1458677 (10kaldari) FYI, this is needed for T188151.
[22:42:53] <wikibugs_>	 10DBA, 10Operations, 10decommission, 10Goal: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#4180499 (10RobH)
[22:58:43] <wikibugs_>	 10DBA, 10Operations, 10decommission, 10ops-eqiad: Decommission db1034 - https://phabricator.wikimedia.org/T182556#4180550 (10RobH)
[22:58:58] <wikibugs_>	 10DBA, 10Operations, 10decommission, 10ops-eqiad: Decommission db1029 and db1031 - https://phabricator.wikimedia.org/T184054#4180551 (10RobH)
[23:00:09] <wikibugs_>	 10DBA, 10Operations, 10decommission, 10ops-eqiad, 10Patch-For-Review: Decommission db1039 - https://phabricator.wikimedia.org/T184262#4180561 (10RobH)
[23:03:16] <wikibugs_>	 10DBA, 10Operations, 10decommission, 10Goal: reclaim and return all cisco servers - https://phabricator.wikimedia.org/T128821#4180580 (10RobH)