[07:03:23] <wikibugs_>	 10DBA, 06Operations, 10ops-codfw: Degraded RAID on db2044 - https://phabricator.wikimedia.org/T159665#3083190 (10Marostegui) 05Open>03Resolved All good now, thanks ```       logicaldrive 1 (3.3 TB, RAID 1+0, OK)        physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 600 GB, OK)       physicaldrive 1I:1:2...
[07:04:42] <wikibugs_>	 10DBA, 06Operations, 10ops-codfw: Degraded RAID on db2048 - https://phabricator.wikimedia.org/T159666#3083192 (10Marostegui) 05Open>03Resolved All good, thanks! ```       logicaldrive 1 (3.3 TB, RAID 1+0, OK)        physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 600 GB, OK)       physicaldrive 1I:1:2 (p...
[07:14:37] <wikibugs_>	 10DBA, 06Operations, 10ops-eqiad, 13Patch-For-Review: Reimage db1060 due to pysical disk corruption (was: Degraded RAID on db1060) - https://phabricator.wikimedia.org/T158193#3083216 (10Marostegui) 05Open>03Resolved Server's original weight has been restored. I will close this ticket, even though there...
[07:18:03] <wikibugs_>	 10DBA, 13Patch-For-Review: Rampant differences in indexes and PK on s6 (frwiki, jawiki, ruwiki) for revision table - https://phabricator.wikimedia.org/T159414#3083221 (10Marostegui) codfw master (db2028) finished correctly: ``` root@neodymium:~# for i in frwiki jawiki ruwiki; do echo $i;mysql --skip-ssl -hdb20...
[07:27:04] <wikibugs_>	 10DBA, 13Patch-For-Review: run pt-table-checksum before decommissioning db1015, db1035,db1044,db1038 - https://phabricator.wikimedia.org/T154485#3083226 (10Marostegui) I have started the checksum on plwiki again after all the crashes with db1060
[07:33:21] <jynus>	 2 rows in set (2 days 16 hours 56 min 5.97 sec)
[07:34:26] <marostegui>	 yep
[07:41:28] <wikibugs_>	 10DBA, 06Operations, 13Patch-For-Review: Install and reimage dbstore1001 as jessie - https://phabricator.wikimedia.org/T153768#3083229 (10Marostegui) I saw no entries on tendril so I assumed the backups that started to run yesterday finished, so I wanted to check if they were ok on bacula, and I saw this: ``...
[10:14:06] <wikibugs_>	 10DBA: Import x1 on dbstore2001 - https://phabricator.wikimedia.org/T159707#3083514 (10Marostegui) In order to start the import I have renamed the tables that aren't supposed to be in use on dbstore2001, that is, all the `echo_%` tables in all the databases but the following: ``` mediawikiwiki metawiki officewik...
[10:14:29] <wikibugs_>	 10DBA: Import x1 on dbstore2001 - https://phabricator.wikimedia.org/T159707#3083516 (10Marostegui) a:03Marostegui
[10:17:56] <wikibugs_>	 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3083531 (10Marostegui) Shall I start copying labsdb1007 to dbstore1001 or are you "breaking" at the moment it @jcrespo?
[10:18:39] <wikibugs_>	 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3083532 (10jcrespo) We can start this now. I was about to do it.
[10:22:30] <wikibugs_>	 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3083581 (10Marostegui) >>! In T157359#3083532, @jcrespo wrote: > We can start this now. I was about to do it.  Ah, go ahead if you like then.
[10:23:54] <wikibugs_>	 10DBA, 06Labs, 13Patch-For-Review: labsdb1004 MySQL crash - https://phabricator.wikimedia.org/T159572#3083591 (10jcrespo) I am going to drop s51412__data from labsdb1004 only , and the others filtered, to avoid confusion on where data is up to date.
[10:29:22] <wikibugs_>	 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3083645 (10jcrespo) I am with labsdb1004 now, please shutdown postgres and copy it.
[10:30:04] <wikibugs_>	 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3083647 (10Marostegui) Oki doki!
[10:37:48] <wikibugs_>	 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3083686 (10Marostegui) Transfer started and the file will be located at: `dbstore1001:/srv/tmp/labsdb1007.tar.gz`
[10:38:15] <jynus>	 what are you copying, all of /srv ?
[10:38:29] <marostegui>	 yes, there is only posgres on /srv/
[10:38:36] <jynus>	 ah, ok
[10:42:36] <wikibugs_>	 10DBA, 06Labs, 13Patch-For-Review: labsdb1004 MySQL crash - https://phabricator.wikimedia.org/T159572#3083690 (10jcrespo) 05Open>03Resolved This is fixed for labsdb1004, except for https://gerrit.wikimedia.org/r/341551 , and labsdb1005, which requires a restart to apply new changes.
[10:47:22] <moritzm>	 jynus: sure, I'll have a look at the socket patch later the day. currently busy with HHVM
[11:11:25] <jynus>	 marostegui, if it is copying and there is not much to do now, I will switch to puppet and monitoring
[11:11:47] <jynus>	 but let me know when it finishes, and I will be on postgres 100%
[11:12:19] <marostegui>	 sounds good!
[11:12:33] <marostegui>	 do you want to do something before reimaging it too?
[11:12:59] <jynus>	 probably copying relevant config on etc and have a look at homes
[11:13:03] <jynus>	 just in case
[11:13:31] <jynus>	 also the list of packages, in case someone has added something unpuppetized
[11:13:33] <marostegui>	 yeah, copying the whole /etc just in case like we did with the other labsd server a couple of weeks ago
[11:13:45] <jynus>	 should be safe, if we do not do both at the same time
[11:13:59] <marostegui>	 yep, no way
[11:15:47] <marostegui>	 just copied /etc
[11:16:33] <wikibugs_>	 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3083756 (10Marostegui) `labsdb1007:/etc` directory has been copied to `dbstore1001:/srv/tmp/labsdb1007_etc.tar.gz`
[11:55:50] <marostegui>	 jynus: the transfer is finished
[11:55:53] <marostegui>	 I also copied /etc
[11:56:14] <marostegui>	 I am going to head for lunch now, feel free to reimage it or whatever you think it is necessary
[11:56:26] <marostegui>	 I took a look at /home and they are all pretty small, so probably nothing importand there
[11:56:41] <jynus>	 ok
[11:57:03] <marostegui>	 maybe also test that we can extract data from the tar.gz?
[11:57:21] <jynus>	 ok
[11:57:31] <marostegui>	 thank you - will be back in a bit
[11:58:36] <marostegui>	 the .tar.gz is 217G and /srv was 945G...
[11:59:19] <marostegui>	 A tar -tfv at least looks good
[11:59:47] <jynus>	 I am decompressing now
[11:59:52] <marostegui>	 ah good
[12:08:10] <wikibugs_>	 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3002535 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by jynus on neodymium.eqiad.wmnet for hosts: ``` ['labsdb1007.eqiad.wmnet'] ```...
[12:15:20] <wikibugs_>	 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3083856 (10jcrespo) The installer ask for verification to delete all partitions- this should be changed on the recipe (or use the db one, where this...
[12:34:18] <wikibugs_>	 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3083907 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['labsdb1007.eqiad.wmnet'] ```  and were **ALL** successful.
[12:40:34] <wikibugs_>	 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3083935 (10jcrespo) It installed ok, but /dev/tank/data has only 500GB. How was that partitioned @akosiaris ?
[12:54:14] <wikibugs_>	 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3083946 (10akosiaris) It's missing the rest of the disks in the md RAID5 (and RAID1 for /boot) for some reason. Maybe some difference between jessie...
[12:55:04] <akosiaris>	 so, somehow that partman recipe failed to add the rest of the disks in the RAID groups. Actually there are no RAID groups from what I see on labsdb1007
[12:56:08] <akosiaris>	 jynus: it only asked about deleting the previous stuff during the install ? nothing else ?
[12:57:34] <akosiaris>	 labsdb100[0-9]|labsdb101[0-1]) echo partman/db.cfg  ?
[12:57:50] <akosiaris>	 line 60 of netboot.cfg ..
[12:58:12] <akosiaris>	 ok that explains it me thinks
[12:58:21] <akosiaris>	 they latter lines 90 and 91 are ignored
[12:59:23] <akosiaris>	 yeah tank/data was never even meant to be present ever in labsdb1007
[13:02:38] <wikibugs_>	 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3083951 (10akosiaris) Nope. not that. What's actually the culprit is rOPUP1b3633e where labsdb1007 gets matched incorrectly
[13:11:08] <wikibugs_>	 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3083958 (10jcrespo) Thanks, I can amend that and try again.
[13:21:43] <marostegui>	 ha, interesting bug!
[13:22:38] <jynus>	 feel free to merge both and try reinstall, I have not finished eating yet
[13:23:37] <marostegui>	 both?
[13:23:51] <marostegui>	 I will merge that patch
[13:25:17] <jynus>	 https://phabricator.wikimedia.org/T157359#3083902
[13:26:00] <marostegui>	 ah oki!
[13:31:47] <wikibugs_>	 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3083998 (10Marostegui) Merged both patches from @jcrespo and ran puppet on install1002, going to try to reimage the server again
[13:33:25] <wikibugs_>	 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3084004 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` ['labsdb1007.eqiad.wmnet']...
[13:42:04] <wikibugs_>	 10DBA, 13Patch-For-Review: Rampant differences in indexes and PK on s6 (frwiki, jawiki, ruwiki) for revision table - https://phabricator.wikimedia.org/T159414#3084034 (10Marostegui) db1061: ``` root@neodymium:~# for i in frwiki jawiki ruwiki; do echo $i;mysql --skip-ssl -hdb1061 $i -e "show create table revisi...
[13:47:11] <jynus>	 I'm back, how is it going?
[13:47:37] <marostegui>	 it got into the console when it just finished installing packages and now got rebooted
[13:47:53] <jynus>	 so it didn't ask for confirmation this time, I assume?
[13:48:05] <marostegui>	 nope, it was finishing the install
[13:48:05] <jynus>	 a.k.a. the #1 patch worked
[13:48:09] <marostegui>	 looks so indeed
[13:48:10] <jynus>	 :-)
[13:48:25] <jynus>	 although that means the one being executed
[13:48:31] <jynus>	 didn't
[13:49:17] <jynus>	 but that was echo partman/db.cfg...
[13:49:26] <jynus>	 which is the one fixed :-/
[13:50:16] <jynus>	 not sure how cases work here, maybe it was trying to do both configs
[13:50:16] <marostegui>	 it is booting up now
[13:51:27] <jynus>	 it takes a lot of time
[13:55:21] <marostegui>	 it is trying to run puppet now
[13:56:59] <marostegui>	 root@labsdb1007:~# pvs
[13:56:59] <marostegui>	   PV         VG            Fmt  Attr PSize PFree
[13:56:59] <marostegui>	   /dev/md1   labsdb1007-vg lvm2 a--  1.64t    0
[13:57:02] <marostegui>	 at least this looks good
[14:06:12] <akosiaris>	 \ο/
[14:07:07] <akosiaris>	 jynus: the cases over there are a first match. and for that reason is very very easy to break. the syntax is particularly unforgiving
[14:07:15] <akosiaris>	 as most bash syntax but still
[14:07:41] <akosiaris>	 but effectively what it does is concatenate multiple small files into a large one and feeding it to d-i
[14:08:12] <jynus>	 is puppet stuck?
[14:08:34] <marostegui>	 no, it is installing postgres now
[14:08:35] <jynus>	 I see it rebooted
[14:08:39] <jynus>	 ah
[14:08:42] <jynus>	 ok, so it takes some time
[14:08:50] <jynus>	 we may need to tune puppet
[14:09:03] <jynus>	 I have not even checked the postgress it will try to install
[14:09:10] <jynus>	 if it is the default or it is hardcoded
[14:09:41] <marostegui>	 let me see
[14:09:44] <jynus>	 it should work, however, in general, beacuse it is the same class than the one used by discovery
[14:10:06] <jynus>	 so they did (or alex did) the work for us in that regard
[14:10:08] <marostegui>	 root      1604  1.9  0.0  68540 33896 ?        Ss   14:00   0:11 /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install postgresql-9.4
[14:10:12] <jynus>	 cool
[14:10:13] <marostegui>	 :)
[14:10:23] <marostegui>	 it is taking quite a while to get it installed
[14:10:24] <jynus>	 you know, the typical
[14:10:26] <marostegui>	 10 minutes now
[14:10:34] <jynus>	 production beta-testing for labs
[14:10:39] <marostegui>	 14:08   0:00 /bin/sh /var/lib/dpkg/info/postgresql-common.postinst configure
[14:10:42] <marostegui>	 haha
[14:10:46] <jynus>	 yeah
[14:10:53] <jynus>	 we will undo most of that
[14:11:05] <jynus>	 that is why we create our own mysql packages
[14:11:28] <jynus>	 to avoid that dangerous debian stuff
[14:12:24] <jynus>	 I would like, however, for nagios and ssh to run first before postgres
[14:13:04] <jynus>	 but both puppet and sql are declarative, witch provides so much fun
[14:13:11] <jynus>	 which
[14:15:55] <marostegui>	 hehe it is starting postgres now
[14:19:55] <jynus>	 it may fail- it may try to replicate from the master?
[14:25:04] <marostegui>	 looks like the service is up at least, puppet keep installing packages finely
[14:29:17] <jynus>	 I keep giving so much s* to mysql, that when alex reminded me of https://eng.uber.com/mysql-migration/ it is not /that/ bad, really
[14:30:56] <akosiaris>	 :-)
[14:42:32] <marostegui>	 Mar  8 14:42:24 labsdb1007 puppet-agent[1013]: Finished catalog run in 2532.21 seconds
[14:42:42] <marostegui>	 I guess the restart will happen soon
[14:42:54] <jynus>	 finally
[14:43:03] <jynus>	 that was a long puppet run
[14:43:49] <wikibugs_>	 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3084272 (10Marostegui) Looks good now: ``` root@labsdb1007:~# pvs   PV         VG            Fmt  Attr PSize PFree   /dev/md1   labsdb1007-vg lvm2 a...
[14:43:57] <marostegui>	 I think it just got restarted
[14:44:30] <jynus>	 yep
[14:47:14] <marostegui>	 it is back now!
[14:47:16] <marostegui>	 \o/
[14:48:12] <wikibugs_>	 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3084282 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['labsdb1007.eqiad.wmnet'] ```  and were **ALL** successful.
[14:51:57] <marostegui>	 so, /srv/postgres isn't mounted
[14:52:10] <marostegui>	 _placeholder labsdb1007-vg -wi-a-----   1.59t that looks like a regex issue 
[14:52:14] <marostegui>	 that _placeholder? :)
[14:54:40] <jynus>	 that seems right based on the recipe
[14:55:21] <jynus>	 other thing is a sed failing or something
[14:56:41] <jynus>	 stat1003, the other host using that recive uses stat1003--vg-srv as a name
[14:57:26] <jynus>	 I do not see, however, any reference to that in a script or puppet
[14:58:09] <marostegui>	 maybe it was renamed manually?
[14:58:32] <jynus>	 you know who can ask?
[14:58:46] <jynus>	 *we
[14:59:09] <marostegui>	 I was checking the last logged people there
[14:59:19] <jynus>	 ?
[14:59:23] <marostegui>	 on that host
[14:59:28] <marostegui>	 so maybe we can ask some of those
[14:59:34] <marostegui>	 otto for instanace
[14:59:40] <jynus>	 random analytics users?
[14:59:46] <jynus>	 I do not think so
[15:00:20] <jynus>	 "_placeholder is an LV that fills up the rest of space."
[15:00:27] <jynus>	 "This is a hack to keep the zookeepeer partition from using up all free space."
[15:00:41] <marostegui>	 aha!
[15:01:33] <jynus>	 I do not mind hacks at all, but they should be documented
[15:02:29] <marostegui>	 we can rename it to srv
[15:08:07] <marostegui>	 done
[15:16:26] <marostegui>	 I have formated that volume with ext3 (same thing it had) and mounted
[15:16:34] <marostegui>	 root@labsdb1007:~# df -hT /srv/
[15:16:35] <marostegui>	 Filesystem                     Type  Size  Used Avail Use% Mounted on
[15:16:37] <marostegui>	 /dev/mapper/labsdb1007--vg-srv ext3  1.6T   69M  1.5T   1% /srv
[15:16:55] <jynus>	 let's remove contents on / first quickly
[15:17:14] <jynus>	  /srv/postgres on / I mean
[15:17:28] <marostegui>	 sure
[15:17:31] <marostegui>	 just umounted it
[15:18:02] <marostegui>	 stopped postgres and moved its content to my home
[15:18:20] <jynus>	 don't start it
[15:18:41] <marostegui>	 no, of course not
[15:20:37] <marostegui>	 when you are ready we can mount it again (mount /srv)
[15:21:27] <jynus>	 yes, I am not doing anything I think
[15:22:50] <marostegui>	 can you do a "pwd"?
[15:22:56] <marostegui>	 are you on /srv/ now?
[15:23:04] <marostegui>	 nevermind mounted
[15:23:35] <jynus>	 let me stop puppet
[15:24:23] <marostegui>	 did you create the content?
[15:24:27] <marostegui>	 it has stuff now
[15:24:36] <jynus>	 puppet may have tried
[15:26:04] <marostegui>	 yeah, there is not much stuff apart from the directories
[15:26:12] <marostegui>	 but if you didn't create them, it was puppet :)
[15:26:44] <jynus>	 let's copy back and install some 9.1 binaries
[15:26:50] <jynus>	 or do as alex said
[15:27:08] <marostegui>	 ok, let's purge those directories again then
[15:27:12] <marostegui>	 and I will start the transfer back
[15:29:32] <marostegui>	 transfer started
[15:29:57] <jynus>	 I have 9.1 on ~/pgsql/bin
[15:30:12] <jynus>	 my ~ not your ~
[15:31:29] <jynus>	 it is 9.1.24
[15:31:48] <jynus>	 the one on ubuntu precise is 9.1.23-0ubuntu0.12.04
[15:32:54] <wikibugs_>	 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3084373 (10Marostegui) So, for the record we saw something with the logical volume: ``` root@labsdb1007:~# lvs   LV           VG            Attr...
[15:32:57] <marostegui>	 oki, I guess it will take another 2 hours or so to copy
[15:33:27] <marostegui>	 we still have the etc backuped, in case we need to check the config files
[15:33:41] <jynus>	 we may
[15:38:25] <wikibugs_>	 10DBA, 06Operations, 13Patch-For-Review: Install and reimage dbstore1001 as jessie - https://phabricator.wikimedia.org/T153768#3084404 (10Marostegui) Good news: the backups were scheduled to run today at 2AM and the first ones started around 45 minutes ago! :-)  So I believe we are in a good shape here, let'...
[16:28:17] <marostegui>	 around 265G in 1 hour, and there were 945….
[16:29:15] <jynus>	 :-/
[16:29:44] <marostegui>	 I guess we will need to continue tomorrow (unless you are planning to stay late)
[16:30:23] <marostegui>	 btw, check your mail
[16:32:32] <jynus>	 yes, I saw it
[16:32:39] <marostegui>	 ok
[17:51:57] <wikibugs_>	 10DBA, 06Labs, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3084728 (10Marostegui) db1070 has been restarted with ROW based replication to serve as a master for db1095.
[18:03:29] <marostegui>	 still 200G to go :(
[18:50:20] <marostegui>	 jynus: transfer is done
[18:50:40] <jynus>	 note sure that last file is complete
[18:51:48] <marostegui>	 it says 100% on both sides, let's give it 10 more minutes
[18:53:48] <jynus>	 let me extract and send 0000000100001B7F00000037 again
[18:54:16] <marostegui>	 ok
[18:54:27] <marostegui>	 i have seen sometimes netcat doing some weird things from a view point of view
[18:55:04] <jynus>	 yes, but normally, that corrupts that last file in my experience
[18:56:53] <marostegui>	 the *0037 has a different size indeed than the rest of the 0003*
[18:57:00] <marostegui>	 but it also has a different owner
[18:57:13] <marostegui>	 very strange
[18:57:40] <jynus>	 yah
[18:57:54] <jynus>	 that is why I do not think it has been transfered correctly
[19:00:14] <marostegui>	 it does have the good owner once it is extraced?
[19:00:25] <jynus>	 it is still extracting
[19:00:30] <marostegui>	 oki
[19:00:41] <jynus>	 tar is not the best of formats
[19:05:56] <marostegui>	 the root thing is strange, I am curious to see how it looks like when you extract it
[19:06:39] <jynus>	 I think tar extracts and then changes the permissions
[19:06:55] <jynus>	 it is likely stuck on the last block or something
[19:09:18] <marostegui>	 shall I crtl+c the transfer and see what it does?
[19:09:30] <jynus>	 sure
[19:09:52] <marostegui>	 it finished
[19:09:55] <marostegui>	 and now looks "good"
[19:10:13] <marostegui>	 owner changed and the file has the same size as the other ones
[19:10:48] <marostegui>	 https://phabricator.wikimedia.org/P5029
[19:11:26] <jynus>	 ok, wanna try the upgrade?
[19:12:00] <marostegui>	 we have to
[19:12:20] <jynus>	 if it breaks, we will have to transfer again
[19:12:29] <marostegui>	 yeah...
[19:12:36] <marostegui>	 you start?
[19:19:17] <jynus>	 I am on it
[19:21:59] <marostegui>	 I have attached myself to the screen to take notes of what you run :)
[19:22:01] <RoanKattouw>	 https://www.irccloud.com/pastebin/njB0gwBa/
[19:22:10] <RoanKattouw>	 A two-year-old row in the recentchanges table?!
[19:22:23] <RoanKattouw>	 Hah there's only two of them
[19:22:31] <jynus>	 that seems like a code bug
[19:22:33] <RoanKattouw>	 https://www.irccloud.com/pastebin/GVQekmqL/
[19:22:38] <jynus>	 it has happened in the past
[19:22:49] <jynus>	 once can be a maintenance bug
[19:23:00] <jynus>	 twice, the code is not reliable
[19:24:19] <RoanKattouw>	 Whoa and rows are disappearing as I watch
[19:24:21] <Reedy>	 probably more than twice if you look across all wikis
[19:24:25] <RoanKattouw>	 Those 2017 rows are gone now
[19:24:42] <RoanKattouw>	 in favor of rows that are a few seconds older
[19:36:32] <RoanKattouw>	 Anyway, the point of that was so I could write https://phabricator.wikimedia.org/T159753#3085157 :)
[19:38:50] <marostegui>	 The doc I was reading also says that the command you are trying to run is the one to upgrade:  pg_upgradecluster 9.1 main
[19:40:07] <marostegui>	 it is listening on 5432
[19:48:01] <wikibugs_>	 10DBA, 10MediaWiki-User-blocking, 03Community-Tech-Sprint: Do test queries for range contributions to gauge performance of using different tables - https://phabricator.wikimedia.org/T156318#3085286 (10MusikAnimal) @jcrespo Any chance you could give this another looksee? Hopefully the one query is enough to i...
[20:06:39] <marostegui>	 is that recovering process something that we need to wait for it to finish before upgrading?
[20:07:03] <jynus>	 I have no I idea what I am doing
[20:07:28] <marostegui>	 I am going thru websites to looking for steps
[20:09:25] <marostegui>	 I was with this: https://scottlinux.com/2015/11/14/upgrade-postgresql-on-debian-jessie-with-pg_upgradecluster/
[20:11:37] <jynus>	 yeah, I tried that, it didn't work
[20:13:06] <marostegui>	 and I was comparing it to: http://karthikeyan2u.blogspot.com.es/2015/07/migrate-postgresql-91-to-94-in-4-steps.html which looks slightly different
[20:13:49] <marostegui>	 the thing is that it still looks like it is trying to start 
[20:13:55] <marostegui>	 or this process is at least confusing
[20:14:02] <marostegui>	 postgres  7189  0.0  0.0 203784  4932 ?        Ss   19:51   0:00 postgres: startup process   recovering 0000000100001B7F00000034
[20:14:07] <jynus>	 it is recovering and it does not start, which is what upgrade does
[20:15:18] <jynus>	 they are only 16MB each, and it is not progressing
[20:18:04] <marostegui>	 yeah, strace says it is timing out all the time
[20:20:09] <marostegui>	 if we stop the 9.4 process, the upgrade complains right?
[20:21:28] <marostegui>	 it is actually doing some processing
[20:21:42] <marostegui>	 postgres: wal receiver process   streaming 1B7F/347E9DE0 
[20:21:45] <marostegui>	 whatever that is
[20:21:51] <marostegui>	 and now
[20:21:58] <marostegui>	 postgres: wal receiver process   streaming 1B7F/347E9D50
[20:22:52] <jynus>	 yeah, it is replicationg
[20:23:04] <jynus>	 I stopped it and it started again
[20:23:55] <jynus>	 maybe dumping is the right way after all
[20:25:41] <marostegui>	 maybe :(
[20:25:50] <marostegui>	 at least that should work no?
[20:28:44] <marostegui>	 sudo -u postgres psql -p 5432
[20:28:46] <marostegui>	 that works
[20:29:44] <jynus>	 yeah, but that is not the one that is failing
[20:34:04] <marostegui>	 why is 5432 listening on 127.0.0.1 anyways?
[20:34:58] <marostegui>	 /home/jynus/pgsql-9.1/bin/postgres -D /srv/postgres/9.1/main -p 54321 -b -c listen_addresses= -> maybe we can try to specify the address it needs to listen to? to see if it forces it?
[20:35:21] <jynus>	 it is not that
[20:35:28] <jynus>	 it would start anyway
[20:35:37] <jynus>	 it starts, it just gets stuck recovering
[20:43:01] <wikibugs_>	 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3085505 (10jcrespo) > Hey, I am not saying it is going to work 100% sure- I am just suggesting to try it first, and then go the slow route, which is...
[20:52:17] <wikibugs_>	 10DBA, 10MediaWiki-User-blocking, 03Community-Tech-Sprint: Do test queries for range contributions to gauge performance of using different tables - https://phabricator.wikimedia.org/T156318#3085518 (10jcrespo) @MusikAnimal The best way to validate the query is to have some sample values (as similar as they w...
[20:53:40] <marostegui>	 I am not completely sure it even starts, as in the port isn't even listening
[20:54:09] <marostegui>	 And I am seegint there is no way to set verbosity or something when it starts so we can try to see something
[20:55:26] <jynus>	 I am going to call for defeat today
[20:56:04] <marostegui>	 yeah
[20:56:31] <marostegui>	 i agreed
[20:56:35] <marostegui>	 agree
[20:58:45] <marostegui>	 i am going to grab dinner
[20:59:02] <marostegui>	 let's chat with alex tomorrow and start the dump process?
[21:34:28] <wikibugs_>	 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3085630 (10MaxSem) Can we use this opportunity to reimport the data from scratch to get rid of possible accumulated OSM replication errors?
[22:14:16] <wikibugs_>	 10DBA, 10MediaWiki-User-blocking, 03Community-Tech-Sprint: Do test queries for range contributions to gauge performance of using different tables - https://phabricator.wikimedia.org/T156318#3085755 (10MusikAnimal) >>! In T156318#3085518, @jcrespo wrote: > @MusikAnimal The best way to validate the query is to...