[05:18:50] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Anti-Harassment, 10Patch-For-Review: Execute the schema change for Partial Blocks - https://phabricator.wikimedia.org/T204006 (10Marostegui) 05Open>03stalled As per T204006#4613570
[07:14:55] <banyek>	 where can I check if the backups are completed on dbstore2002?
[07:15:03] <banyek>	 (good morning btw)
[07:15:04] <marostegui>	 Hi!
[07:15:22] <marostegui>	 jynus and myself just saw that we have s2 duplicated on both, dbstore2001 and dbstore2002
[07:15:28] <marostegui>	 being dbstore2001 the active source
[07:17:06] <banyek>	 👍
[07:17:27] <marostegui>	 banyek: that probably means we can remove the data from dbstore2002 entirely
[07:17:36] <banyek>	 OH
[07:17:49] <jynus>	 I am not so sure about that
[07:17:56] <marostegui>	 tendril down?
[07:17:56] <jynus>	 I would keep the one on dbstore2002
[07:18:08] <jynus>	 tendril seems down to me indeed
[07:18:12] <banyek>	 tendril works for me
[07:18:23] <marostegui>	 we just got paged
[07:18:49] <banyek>	 I db1115 us for tendril?
[07:19:01] <banyek>	 I look after this
[07:20:26] <marostegui>	 banyek: check -operations channel
[07:20:28] <marostegui>	 we are debugging there
[07:20:30] <banyek>	 there are plenty of space but you know it, I see you logged in :D
[07:50:10] <wikibugs>	 10DBA: db1092 crashed - https://phabricator.wikimedia.org/T205514 (10Marostegui)
[07:51:19] <jynus>	 something happened at 7:16 at tendril
[07:51:53] <jynus>	 https://grafana.wikimedia.org/dashboard/db/prometheus-machine-stats?orgId=1&var-server=db1115&var-datasource=eqiad%20prometheus%2Fops&from=now-6h&to=now-1m
[07:52:36] <marostegui>	 Maybe db1092 being unavailable triggered something on tendril? some sort of overload?
[07:53:33] <jynus>	 many things could be
[07:53:35] <wikibugs>	 10DBA: db1092 crashed - https://phabricator.wikimedia.org/T205514 (10Marostegui) On reboot: ``` 313-HPE Smart Storage Battery 1 Failure - Battery Shutdown Event Code: 0x0400. Action: Restart system. Contact HPE support if condition persists. ```
[07:53:39] <jynus>	 when did db1092 go down?
[07:54:07] <jynus>	 at 6:10
[07:54:07] <marostegui>	 as per its HW logs, the failure is at 06:05
[07:54:24] <jynus>	 6:11:32 was the last prometheus report
[07:54:58] <jynus>	 could be some build up, sure, but that is true with almost anything- memory leak, etc.
[07:55:14] <jynus>	 but nprd didn't work at all, aparently
[07:56:07] <jynus>	 it starte alerting at 7:17
[07:57:08] <jynus>	 didn't banyek wanted to practice cloning? he will be able to do it with db1092 :-D
[07:57:17] <marostegui>	 indeed :)
[07:57:18] <banyek>	 cool then
[07:57:24] <marostegui>	 I am upgrading kernel and mariadb and giving it a last reboot
[07:57:26] <jynus>	 although let's wait for
[07:57:31] <jynus>	 full debugging
[07:57:36] <banyek>	 We start to upgrade db1107 now, after that I can reclone it
[07:57:48] <jynus>	 no rush for db1092
[07:57:55] <jynus>	 altough let's depool it
[07:58:03] <marostegui>	 As long as we reclone it before the failback, I am fine
[07:58:08] <marostegui>	 jynus: I am doing the patch now
[07:58:20] <jynus>	 I will disable alerts
[07:58:51] <banyek>	 FYI: my daugther is sick again - today my wife is here, but possibly tomorrow and friday I have to deal with her
[07:59:03] <banyek>	 at least until 2 or something
[07:59:09] <banyek>	 I try to figure out something
[08:06:24] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1092 crashed - https://phabricator.wikimedia.org/T205514 (10Marostegui) p:05Triage>03Normal a:03Cmjohnson @Cmjohnson looks like we need a new BBU. This host is under warranty, can you talk to HP and see if we can get a new BBU before 10th Oct (a...
[08:15:02] <wikibugs>	 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Wikidata-Campsite, 10User-Ladsgroup: Schema change for adding indexes of ct_tag_id - https://phabricator.wikimedia.org/T203709 (10Marostegui)
[08:15:31] <jynus>	 marostegui: can you see why I disable lagging alerts on dbstores? they keep spaming -operations
[08:16:19] <marostegui>	 indeed :)
[08:16:23] <marostegui>	 I didn't enable them eh!
[08:19:16] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1092 crashed - https://phabricator.wikimedia.org/T205514 (10Marostegui)
[08:19:25] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1092 crashed - BBU broken - https://phabricator.wikimedia.org/T205514 (10Marostegui)
[08:21:07] <wikibugs>	 10DBA: BBU problems dbstore2002 - https://phabricator.wikimedia.org/T205257 (10Marostegui) Thanks @Papaul! So my proposal is to do: T205257#4610104 @jcrespo @Banyek thoughts on that?
[08:21:26] <wikibugs>	 10DBA, 10User-Banyek: Maintenance M4 cluster - https://phabricator.wikimedia.org/T205288 (10Banyek) db1107 upgrade done, the operaton is resumed
[08:22:35] <wikibugs>	 10DBA, 10User-Banyek: Maintenance M4 cluster - https://phabricator.wikimedia.org/T205288 (10Marostegui) Great! Make sure to update: https://phabricator.wikimedia.org/P7510 and reload haproxy!
[08:22:42] <wikibugs>	 10DBA, 10User-Banyek: Maintenance M4 cluster - https://phabricator.wikimedia.org/T205288 (10Banyek) upgrade of db1108 is planned on 2018-09-27 10:00 CEST
[08:24:28] <jynus>	 marostegui: the controlled failed, not only the battery
[08:25:03] <marostegui>	 really? the idrac logs didn't say that, only mention battery
[08:25:11] <jynus>	 I will show you in a sec
[08:25:15] <jynus>	 when they finish downloading
[08:25:20] <marostegui>	 Great!
[08:25:22] <jynus>	 pasting to ticket
[08:25:30] <marostegui>	 Too bad they have different info on the idrac than on the web interface
[08:28:30] <wikibugs>	 10DBA, 10User-Banyek: Maintenance M4 cluster - https://phabricator.wikimedia.org/T205288 (10Banyek) Update for 1108: db1108      Agree in a usable maintenance window     Downtime dbproxy1004 and dbproxy1009 services in icinga     Disable eventogging_sync.sh on db1108     Disable eventlogging_cleaner     Do the...
[08:31:29] <banyek>	 is there any reason why the haproxies don't have the /stats webui enabled?
[08:32:07] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1092 crashed - BBU broken - https://phabricator.wikimedia.org/T205514 (10jcrespo) {F26209361}  AHS log on google drive, cannot attach it on phabricator: https://drive.google.com/open?id=1Y9RikXhRlZHY-7gNN0MeK5AdOjswPXJH
[08:33:10] <jynus>	 ^ marostegui: I was wrong, the power loss is old
[08:33:21] <marostegui>	 So only BBU then?
[08:33:27] <jynus>	 yep
[08:33:44] <marostegui>	 I cannot download that file
[08:33:45] <jynus>	 and it was like the last time- a disk degrading brought down the whole thing
[08:33:56] <jynus>	 which one?
[08:34:02] <marostegui>	 the one in gdrive
[08:34:12] <jynus>	 that is shared with all WMF accoutns
[08:34:27] <marostegui>	 yeah, but gdrive does nothing
[08:34:27] <jynus>	 does it fail or say permission denied?
[08:34:33] <jynus>	 let me check
[08:34:36] <marostegui>	 no, it just doesn't download
[08:35:40] <jynus>	 try now
[08:35:43] <marostegui>	 lets see
[08:35:56] <jynus>	 it is not as if you are going to read anything
[08:36:07] <marostegui>	 it keeps scanning for viruses...
[08:36:19] <marostegui>	 now it worked
[08:36:22] <marostegui>	 what did you do?
[08:36:51] <jynus>	 changed the link
[08:36:55] <marostegui>	 haha
[08:36:58] <marostegui>	 It works now
[08:37:49] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1092 crashed - BBU broken - https://phabricator.wikimedia.org/T205514 (10Marostegui)
[08:42:09] <elukey>	 hello :) Time to chat now about dbstore1002 goals or an email is betteR?
[08:43:04] <marostegui>	 elukey: give me 10 mins if you can
[08:45:15] <elukey>	 ack!
[08:48:52] <wikibugs>	 10DBA, 10User-Banyek: dbstore2002 tables compression status check - https://phabricator.wikimedia.org/T204930 (10Banyek) compressing s2 is resumed
[08:56:58] <marostegui>	 elukey: I am back!
[08:59:09] <marostegui>	 banyek: I am not planning to debug db1092 any further, as we have everything we need. Not sure if jynus wants to do something with it, but if not, we can reclone it if you like. I doubt chris will get to it today, as he needs first to deal with HP
[09:00:33] <banyek>	 jynus:? ^^
[09:06:15] <jynus>	 ok
[09:06:31] <jynus>	 but shouldn't we talk about backupd first?
[09:06:37] <banyek>	 we should
[09:06:47] <banyek>	 I'd prefer to something more important 
[09:06:51] <banyek>	 *do
[09:07:05] <marostegui>	 backupd?
[09:07:08] <jynus>	 db1092 is going to be there today and tomorrow
[09:07:12] <jynus>	 backups
[09:07:24] <marostegui>	 ah
[09:07:37] <jynus>	 I use a qwerty keyboard where s and d are close toghether
[09:07:40] <marostegui>	 sure, whatever you guys want
[09:07:47] <elukey>	 marostegui: ack! So I had a chat with my team about a possible goal, and we thought that we could split it into two parts. The next Q should be dedicated to get the hardware, set it up and possibly (maybe stretch goal?) to have puppet config deployed and data replicated. Then in another quarter the replacement of research user (with a new way to provide accounts) and the migration from dbstore1
[09:07:53] <elukey>	 002 to the new system. Does it sound acceptable for you guys?
[09:08:06] <marostegui>	 jynus: No need to explain that, I use the same keyboard. I thought backupd was some sort of daemon or something
[09:08:13] <jynus>	 :-)
[09:09:38] <marostegui>	 elukey: you are not planning to order the hardware during Q2?
[09:09:49] <marostegui>	 elukey: with next quarter you mean the one starting next week?
[09:10:23] <jynus>	 wow, you use the same keyboard than me!? you mean like https://youtu.be/1Y2zo0JN2HE?t=12s ?
[09:11:01] <marostegui>	 jynus: I hope you use a mechanical one, otherwise I am not talking to you anymore
[09:11:10] <elukey>	 marostegui: yes next quarter I mean Q2, in theory the hardware should already been in the pipeline to be ordered IIUC, procurement tasks etc.. are already in place waiting for a quote
[09:12:12] <jynus>	 well, mid-Q3 the server goes down, we have a replacement or not, you heard faidon
[09:12:20] <marostegui>	 elukey: Then yeah, getting the hardware and trying to get puppet adapted for dbstore1002-multiinstance sounds like a good idea. If that is done on time, we can help with getting all the data there
[09:12:28] <marostegui>	 And obviously advise on the puppet stuff
[09:13:24] <elukey>	 jynus: yep exactly, but at the same time it seems a lot to set up the replacement and migrate user (with new accounts) in one Q, given the fact that you guys are already super full
[09:13:42] <elukey>	 marostegui: I have no idea about what puppet work needs to be done but I can surely help/work on it
[09:13:46] <jynus>	 well, we are not in charge of that
[09:13:49] <marostegui>	 elukey: I would assume your team would take the lead on the HW orderning + puppet?
[09:13:53] <jynus>	 analytics is
[09:14:00] <jynus>	 we can can just help you do it!
[09:15:52] <elukey>	 jynus: sure but up to now you guys have managed it, and I will surely not able to do maintenance to three new databases with my limited knowledge :D. So I agree that technically we own it (as analytics), but practically I'd need some guidance from  you about how to set up this work correctly
[09:16:11] <marostegui>	 elukey: that is fine, we can help with that
[09:16:12] <jynus>	 for the 3rd time, we will help
[09:16:18] <jynus>	 4 time now :-)
[09:16:52] <elukey>	 and for the 4th time I am telling you the same thing, since you keep stating that "analytics owns it" :)
[09:17:08] <elukey>	 so we are kinda repeating the same thing each time on both sides
[09:17:22] <marostegui>	 elukey: My main question for you (your team) if you'll lead that. The orderning process (we can help reviewing the quotes), following up the racking and installation and getting some puppet code up (so we can review and advise)
[09:18:40] <elukey>	 marostegui: sure I can definitely lead it
[09:19:40] <marostegui>	 elukey: That's good. So maybe we need to agree on the scope now. I think getting the hardware ordered + racked + installed sounds like a good idea. Not completely sure about the data and puppet part
[09:19:45] <marostegui>	 Maybe that can be stretch
[09:20:04] <elukey>	 exactly, I think it is good
[09:20:29] <elukey>	 I'll try to push for the stretch anyway since, as Jaime was saying, mid Q3 seems really near :D
[09:20:31] <marostegui>	 elukey: As you said we're pretty full already + holidays + xmas time
[09:20:58] <marostegui>	 elukey: Yeah, I am fine with having it as a stretch point
[09:21:45] <elukey>	 ok then, I am going to explain what we discussed in here and ask Nuria/Mark to review the goals accordingly
[09:21:46] <marostegui>	 elukey: My main concern is not adapting multinstance role that we already have to dbstore1002, but discovering all the stuff that is hardcoded in dbstore1002 that we might not be even aware of :)
[09:22:16] <elukey>	 it will be fun! 
[09:22:37] * marostegui has another concept of "fun"
[09:28:28] <banyek>	 https://www.youtube.com/watch?v=dMQdqO4RV3w
[09:28:49] <volans>	 lol
[09:32:09] * elukey avoids to link Rebecca Black - Friday in here
[09:32:51] <jynus>	 FRIDAY FRIDAY!!!!!!
[09:33:17] <marostegui>	 http://isitfridayyet.net/
[09:33:20] <marostegui>	 :(
[09:42:48] <jynus>	 I am going to deploy the new root client grants to all databases
[09:42:55] <marostegui>	 ok
[10:03:47] <marostegui>	 I thought we changed the labs passwords
[10:04:10] <marostegui>	 don't they have a different one from core?
[10:04:23] <jynus>	 we did
[10:04:32] <jynus>	 but I applied the new ones with the same one
[10:04:36] <jynus>	 so I have to undo that
[10:04:41] <marostegui>	 aah right
[10:05:03] <jynus>	 and that is why we need a tool
[10:05:27] <jynus>	 then there is some hosts that failed that shouldn't have
[10:05:50] <moritzm>	 sorry. there's also a Juniper change pending, maybe that affected labsdb? https://phabricator.wikimedia.org/T205513
[10:05:53] <jynus>	 like db1107
[10:06:02] <jynus>	 no, nothing to do, they don't failed
[10:06:05] <moritzm>	 ok
[10:06:38] <jynus>	 so db1107 doesn't have the proper grants
[10:14:54] <banyek>	 I go to have some lunch
[10:15:18] <jynus>	 elukey: I have uniformized the grants on eventlogging dbs
[10:15:38] <jynus>	 they had some unsecure defaults
[10:15:43] <jynus>	 for root
[10:17:18] <elukey>	 ah thanks! Can I know the grants just as FYI? (curious, but when you have time)
[10:17:44] <jynus>	 the old ones or the new ones?
[10:17:57] <jynus>	 it had a root@10.x
[10:18:14] <jynus>	 I have restricted to the root clients and to the 2 eventlogging hosts
[10:18:34] <jynus>	 and removed the grant option from the eventlogging hosts
[10:19:15] <jynus>	 eventlogging sync was temp. halted
[10:19:23] <jynus>	 but then I checked to put it back running
[10:20:05] <jynus>	 db1107 wasn't available from neodymium/cuminXXXX etc
[10:20:29] <jynus>	 more details of the fixes at T177385
[10:20:30] <stashbot>	 T177385: Upgrade Cumin masters to stretch - https://phabricator.wikimedia.org/T177385
[10:21:53] <jynus>	 marostegui: for fun, run "pt-show-grants | grep root" on db1118
[10:23:18] <marostegui>	 it doesn't work?
[10:23:25] <jynus>	 IT DOES
[10:23:28] <jynus>	 too well
[10:23:46] <marostegui>	 Ah, i was running that exact string you sent
[10:23:50] <jynus>	 I may have broken it
[10:23:50] <marostegui>	 let me get it to connect :)
[10:24:01] <jynus>	 I am precisely changing root@localhsot
[10:24:06] <marostegui>	 that same command doesn't work now
[10:24:15] <marostegui>	 what was the issue?
[10:24:41] <jynus>	 it was a very strange format
[10:24:49] <jynus>	 I may have killed db1118 access
[10:25:19] <marostegui>	 Who neds mysql 8.0 anyways!
[10:25:22] <marostegui>	 *needs
[10:25:22] <jynus>	 or it doesn't work with the mariadb client
[10:26:42] <jynus>	 I am going to try to make it work
[10:27:00] <marostegui>	 :)
[10:28:41] <jynus>	 mysql --disable-auto-rehash --ssl-mode=DISABLED --socket=/run/mysqld/mysqld.sock
[10:28:49] <jynus>	 it just needs tweaking
[10:29:00] <jynus>	 and the using sock_auth is not immediately apparent on the grants
[10:32:10] <jynus>	 yeah, our grant commands don't work on mysql 8
[10:34:16] <wikibugs>	 10DBA, 10Operations, 10Epic, 10Patch-For-Review: DB meta task for next DC failover issues - https://phabricator.wikimedia.org/T189107 (10Marostegui) An update on this. We have pretty much completed all the tasks we had scheduled for the failover. We are now advancing on other tasks, to complete them faster...
[10:40:20] <jynus>	 there is a lot to change with mysql: grant and account handling is different
[10:40:28] <jynus>	 and user table is different too
[10:44:14] <marostegui>	 yeah, I was wondering about the user table indeed
[10:44:23] <marostegui>	 I haven't checked it but I was wondering if it was too different
[10:54:35] <jynus>	 there is 2 bugs on mysql.py
[10:54:46] <jynus>	 one is that we don't correctly handle .wikimedia hosts
[10:55:18] <jynus>	 the other is that I just realized I don't add /root/.my.cnf as the deafault place too look configuration
[10:55:25] <jynus>	 so either it requires sudo -i
[10:55:40] <jynus>	 or a change to the mysql command line arguments
[10:56:44] <banyek>	 breferably both I would say
[12:32:56] <marostegui>	 banyek: what happened with the options you set on dbstore2002 to ease replication? did you revert them?
[12:57:02] <wikibugs>	 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Wikidata-Campsite, 10User-Ladsgroup: Schema change for adding indexes of ct_tag_id - https://phabricator.wikimedia.org/T203709 (10Marostegui)
[13:51:23] <wikibugs>	 10DBA, 10User-Banyek: dbstore2002 tables compression status check - https://phabricator.wikimedia.org/T204930 (10Banyek) In T204593 I was mentioned that I reenabled the cache on the host even the BBU is broken - I disabled it because of the hosts SPOF-ness
[14:31:20] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1092 crashed - BBU broken - https://phabricator.wikimedia.org/T205514 (10Cmjohnson) A support ticket has been submitted with HPE  Case ID: 5332806955
[14:31:46] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1092 crashed - BBU broken - https://phabricator.wikimedia.org/T205514 (10Marostegui) Thanks!! :-)
[15:11:13] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Banyek: db1092 crashed - BBU broken - https://phabricator.wikimedia.org/T205514 (10Banyek) I will reclone this database instance
[15:12:06] <wikibugs>	 10DBA, 10MediaWiki-extensions-Translate, 10Operations, 10Performance-Team, and 2 others: DBPerformance warning "Query returned 22186 rows: query: SELECT * FROM `translate_metadata`" on Meta-Wiki - https://phabricator.wikimedia.org/T204026 (10Gilles)
[15:41:42] <banyek>	 Ok folks I guess I'll leave for today
[15:42:57] <addshore>	 marostegui: is it possible for me to look at what is in the parsercache servers?
[15:43:31] <addshore>	 I guess thats the only way for me to judge the impact on them, is to see what is in them :)
[15:43:34] <jynus>	 addshore: only the mediawiki wise men can interpreta that format
[15:43:51] * addshore isn't even sure if he has access to there
[15:43:54] <jynus>	 as in, probably, but you are asking in the wrong channel, for us that is a black box
[15:44:14] <addshore>	 okay!
[15:44:29] <jynus>	 also, if you want to see the state, mostly you want to check the memcache + parsercache combination
[15:44:36] <Reedy>	 you should be able to sql dbname -h db12345
[15:44:43] <Reedy>	 or whatever the new syntax is
[15:45:08] <jynus>	 Reedy: he said look what is in read, not just SELECT it, that may need some code
[15:45:47] <jynus>	 only the arcane ones know how to read parsercacke keys!
[15:46:27] <jynus>	 or soemting that calls some obscure mediawiki internal function :-)
[15:47:04] <jynus>	 Reedy: sorry, you were answering the access part, yes, that has the same credentials than the otehr dbs
[15:47:18] <Reedy>	 :)
[15:47:28] <jynus>	 although it is configured by ip, so not 100% it will work
[15:47:56] <Reedy>	 it definitely did at one point in the past
[15:48:07] <Reedy>	 But not tried anytime recently
[15:48:36] <jynus>	 you need to put 3 candles to the mediawiki devil and sacrifice a goat for it to work now!
[15:48:54] <Reedy>	 The germans have a lot of goats
[15:49:15] <jynus>	 the germans have a lot of Qs in identifiers!
[15:58:38] <addshore>	 Reedy: hmm, struggling to access..
[15:59:08] <jynus>	 see, how many goats, only 1, 2?
[15:59:13] <addshore>	 aaah, --host not -h
[15:59:20] <jynus>	 lol
[16:00:41] <addshore>	 perhaps I need to specify a "cluster" or "group"
[16:00:51] <addshore>	 i guess perhaps a cluster, not a group
[16:00:58] <jynus>	 pc1/pc2/pc3?
[16:01:12] <jynus>	 it is a special set of hosts because they don't contain data
[16:01:25] <jynus>	 you should be able to get the pass and query the ips directly
[16:01:33] * addshore was trying pc2004 or the ips related to that
[16:01:43] <jynus>	 please be careful, they are in read-write!
[16:01:50] <jynus>	 unlike most metadata hosts
[16:01:53] * addshore will be :)
[16:03:17] <jynus>	 also, don't say I didn't warn you about that you may be disappointed about what you will find there
[16:05:32] <addshore>	 Reedy: I think your tips are missleading, you made it seem easy
[16:06:36] <jynus>	 did you manage it?
[16:06:55] <addshore>	 as far as I can tell you can't use sql.php
[16:07:02] <jynus>	 :-(
[16:07:03] * addshore will try with mysql now
[16:07:11] <jynus>	 you should be able to do it manually
[16:07:55] <jynus>	 you may think I am not telling you how, but I genunily don't know, never touch mediawiki or the mediawiki acounts at all
[16:08:17] <jynus>	 we manage the dbs off-band with a separate set of accounts
[16:08:55] <addshore>	 ahh, sql.php actually allows you to pass parameters through to the mysql command, then then still do its magic for fetching dbnames!
[16:09:28] * addshore might have to tap out and try this again later
[16:10:23] <Reedy>	 ERROR 1045 (28000): Access denied for user 'wikiuser'@'10.64.32.16' (using password: YES)
[16:10:34] <addshore>	 different user?
[16:10:36] <jynus>	 to be fair, not accessing the pc hosts may be a feature
[16:10:38] <addshore>	 anyway, im off for now
[16:10:49] <jynus>	 intended by the sql.php auther
[16:10:58] * addshore still needs to try to purge the parser cache up to the current time for wikidata tonight :)
[16:11:54] <addshore>	 Reedy: if you find out how feel free to send my way! :)
[16:12:12] <jynus>	 Reedy: you may be mixing passwords
[16:12:17] <jynus>	 that is certainly possible
[16:13:39] <Reedy>	 I am
[16:13:50] <Reedy>	 mysql:wikiadmin@pc1004 [(none)]> 
[16:13:54] <Reedy>	 Yeah, right password helps :P
[16:16:42] <Reedy>	 Is wgDBsqlpassword even still used/
[16:17:06] <Reedy>	 https://www.mediawiki.org/wiki/Manual:$wgDBsqlpassword
[16:17:12] <Reedy>	 	This feature was removed from MediaWiki core in version 1.5.0.
[16:17:16] <Reedy>	 Why do we still have it set...
[16:20:39] <addshore>	 Ooo, you got in though?
[16:40:42] <addshore>	 Reedy: ^^
[17:05:09] <Reedy>	 yeah
[17:23:40] <addshore>	 With what command? :)
[17:31:30] <Reedy>	 mysql -u wikiadmin -h pc1004 -p
[18:28:13] <wikibugs>	 10DBA, 10MediaWiki-Watchlist, 10Growth-Team (Current Sprint), 10Wikimedia-production-error: Deleting large watchlist takes > 4 seconds causing rollback due to write time limit - https://phabricator.wikimedia.org/T171898 (10JTannerWMF)