[00:04:06] <grrrit-wm>	 (03PS2) 10Legoktm: Add #releng to #-qa [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/175567 (owner: 10Greg Grossmeier)
[00:05:16] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032 V: 032] Add #releng to #-qa [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/175567 (owner: 10Greg Grossmeier)
[00:05:34] <grrrit-wm>	 (03PS2) 10Legoktm: Renamed some phab projects, so renaming them here [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/175564 (owner: 10Greg Grossmeier)
[00:05:58] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032 V: 032] Renamed some phab projects, so renaming them here [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/175564 (owner: 10Greg Grossmeier)
[00:05:59] <wikibugs>	 3Beta-Cluster, Wikimedia-Labs-Infrastructure: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - https://phabricator.wikimedia.org/T71601#783100 (10greg)
[00:06:13] <wikibugs>	 3Beta-Cluster, Wikimedia-Labs-Infrastructure: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - https://phabricator.wikimedia.org/T71601#727373 (10greg)
[00:07:01] <legoktm>	 !log restarting wikibugs for -qa changes
[00:07:02] <labs-morebots>	 restarting is not a valid project.
[00:07:08] <legoktm>	 !log tools.wikibugs restarting wikibugs for -qa changes
[00:07:11] <labs-morebots>	 Logged the message, Master
[01:38:02] <comets>	 you guys broke oauth it seems, keeps requesting authentication every time u try to use one of the tools..
[10:06:01] <wikibugs>	 3Tool-Labs-tools-Other: [tracking] toolserver.org tools that have not been migrated - https://phabricator.wikimedia.org/T60865#783806 (10Magnus)
[10:06:02] <wikibugs>	 3Tool-Labs-tools-Erwin's-tools: Migrate https://toolserver.org/~erwin85/contribs.php to Tool Labs - https://phabricator.wikimedia.org/T62870#783803 (10Magnus)
[10:07:47] <YuviPanda>	 !log analytics updated labs/private manually, forcing puppet run on wikimetrics-dev1
[10:07:51] <labs-morebots>	 Logged the message, Master
[10:08:54] <wikibugs>	 3Tool-Labs-tools-Erwin's-tools: Migrate https://toolserver.org/~erwin85/contribs.php to Tool Labs - https://phabricator.wikimedia.org/T62870#783817 (10Liuxinyu970226)
[11:04:32] <wikibugs>	 3Wikimedia-Labs-Infrastructure: Labs: Enable "Puppet freshness" checks in shinken for cvn project - https://phabricator.wikimedia.org/T68573#783932 (10yuvipanda)
[11:06:12] <YuviPanda>	 !log shinken ln -s /bin/true /usr/bin/mail on shinken-server-01 to prevent it from spamming people same time shinken-01 does. See https://phabricator.wikimedia.org/T75859
[11:06:14] <labs-morebots>	 Logged the message, Master
[11:24:48] <YuviPanda>	 !log shinken made /usr/bin/mail a small script that just logs to /data/project/shinken-server-01.email on shinken-server-01
[11:24:50] <labs-morebots>	 Logged the message, Master
[11:42:14] <YuviPanda>	 Krinkle|detached: ping when around, I'm going to migrate icinga checks for integration into shinken, and can setup alerts for cvn too now
[11:54:00] <Merlissimo>	 is there a list to which project a bug should be assigned? e.g. for a dns problem Labs-team wikimedia-labs-infrastructure wikimedia-labs-tools
[11:55:22] <wikibugs>	 3Labs-Team, Wikimedia-Labs-Infrastructure, Wikimedia-Labs-tools: maiwiki: s3.labsdb server alias maiwiki.labsdb missing in /etc/hosts and wrong name value in meta__p - https://phabricator.wikimedia.org/T75861 (10Merl) 3NEW p:3High a:3coren
[12:15:39] <wikibugs>	 3Wikimedia-Labs-Infrastructure, Labs-Team: Have shinken check for basic labs infrastructure - https://phabricator.wikimedia.org/T75865 (10yuvipanda) 3NEW p:3Triage
[12:28:14] <dan-nl>	 i don't see a chat room for discussing phabricator issues. is there one? specifically i'd like to find out what i'm doing wrong in regards to logging into phabricator
[12:29:59] <YuviPanda>	 dan-nl: #wikimedia-devtools
[12:30:12] <dan-nl>	 thanks YuviPanda 
[12:42:36] <wikibugs>	 3Wikimedia-Labs-General: Rename project bots to wm-bot - https://phabricator.wikimedia.org/T57691#784099 (10Petrb) p:5Triage>3Volunteer?
[14:35:53] <tonythomas_>	 what is the hostname of http://deployment.wikimedia.beta.wmflabs.org/ again ?
[14:36:07] <tonythomas_>	 deployment-mediawiki02 ?
[14:37:28] <tonythomas_>	 !log removed instance mediawiki-verp due to stale puppet status
[14:37:29] <labs-morebots>	 removed is not a valid project.
[14:38:07] <tonythomas_>	 !log mediawiki-verp removed instance mediawiki-verp due to stale puppet status
[14:38:11] <labs-morebots>	 Logged the message, Master
[14:38:24] <tonythomas_>	 yay ! that was a first :) 
[14:41:18] <tonythomas_>	 err refreshing https://wikitech.wikimedia.org/wiki/Special:NovaInstancewill create a new instance :\ again and again 
[14:55:06] <wikibugs>	 3Tool-Labs-tools-Other: Encoding problems on IRC logs due to IRC color codes - https://phabricator.wikimedia.org/T71514#784480 (10Petrb) I am wondering why don't you just html interface http://tools.wmflabs.org/wm-bot which display these as colors I don't see any point in removing these from txt files as some pr...
[15:41:58] <wikibugs>	 3Labs-Team, Wikimedia-Labs-Infrastructure, Wikimedia-Labs-tools: maiwiki: s3.labsdb server alias maiwiki.labsdb missing in /etc/hosts and wrong name value in meta__p - https://phabricator.wikimedia.org/T75861#784605 (10coren)
[15:42:43] <wikibugs>	 3Labs-Team, Wikimedia-Labs-Infrastructure, Wikimedia-Labs-tools: maiwiki: s3.labsdb server alias maiwiki.labsdb missing in /etc/hosts and wrong name value in meta__p - https://phabricator.wikimedia.org/T75861#784606 (10coren) 5Open>3Resolved
[15:55:24] <wikibugs>	 3Wikimedia-Labs-tools: Dumps not updating again. - https://phabricator.wikimedia.org/T74154#784621 (10coren) 5Open>3stalled
[15:59:18] <wikibugs>	 3Wikimedia-Labs-tools: Dumps not updating again. - https://phabricator.wikimedia.org/T74154#784630 (10coren)
[16:00:02] <wikibugs>	 3Wikimedia-Labs-tools: sql script does not accept wildcards as parameter - https://phabricator.wikimedia.org/T75595#784631 (10coren)
[16:00:04] <wikibugs>	 3Wikimedia-Labs-tools: Clean out files in /tmp older than 5 days - https://phabricator.wikimedia.org/T75527#784632 (10coren)
[16:00:26] <Coren>	 Holey.  Is this going to spam the channel like this every change of column?  Because I have >90 to move.
[16:00:46] <YuviPanda>	 Coren: poke legoktm
[16:00:51] <YuviPanda>	 or valhallasw`cloud
[16:01:17] <valhallasw`cloud>	 Coren: muting workboard columns moves is somewhere on the to do list...
[16:01:37] <valhallasw`cloud>	 so I'd suggest just muting wikibugs here while moving
[16:01:49] <Coren>	 kk
[16:02:03] <valhallasw`cloud>	 sorry about that
[16:05:15] <wikibugs>	 3Wikimedia-Labs-tools: DB replication results are slow. - https://phabricator.wikimedia.org/T75420#784641 (10coren)
[16:05:23] <valhallasw`cloud>	 it's wikibugs, not wm-bot ;-)
[16:05:28] <Coren>	 Oh duh.  wikibugs not wm-bot
[16:05:39] <YuviPanda>	 haha
[16:17:33] <tonythomas_>	 YuviPanda: http://deployment.wikimedia.beta.wmflabs.org/ == deployment-mediawiki02 right ?
[16:17:55] <YuviPanda>	 hmm, no idea, tonythomas_...
[16:17:58] <YuviPanda>	 it probably is not
[16:18:05] <YuviPanda>	 I bet there's varnishes and etc behind that
[16:18:18] <tonythomas_>	 which is that one then ? bd808 any idea ?
[16:18:21] <YuviPanda>	 this is also the first time I've seen the hostname
[16:18:29] <YuviPanda>	 it points to 208.80.155.135
[16:18:36] <tonythomas_>	 oh 
[16:18:38] <JohnFLewis>	 tonythomas_: I can tell you it is not -03 :)
[16:18:44] <YuviPanda>	 https://wikitech.wikimedia.org/w/api.php?action=query&list=novainstances&niproject=deployment-prep&niregion=eqiad&format=json
[16:18:49] <YuviPanda>	 it's deployment-cache-text02
[16:18:50] <bd808>	 -01 and 02 are behind the same varnish
[16:19:19] <tonythomas_>	 so - its deployment-cache-text02 ?
[16:19:20] * YuviPanda finds the API faster and easier to use than Special:NovaInstances
[16:19:24] <bd808>	 Requests are round-robin balanced between them
[16:19:41] <tonythomas_>	 bd808: between cache-text02 and mediawiki-02 ?
[16:20:06] <bd808>	 mediawiki01 and mediawiki02 are behind all of the varnishes as the MW servers
[16:20:18] <bd808>	 so the requests ultimately end up there
[16:21:13] <tonythomas_>	 Jeff_Green: looks like we can go with the exim logs in mediawiki-02, ( I think so ) 
[16:23:32] <bd808>	 YuviPanda: We need a cute icon for labs-vagrant -- https://phabricator.wikimedia.org/tag/labs-vagrant/
[16:23:59] <YuviPanda>	 let me poke isarra, maybe she'll make an ascii unicorn
[16:24:14] <bd808>	 Oh. we have one in the wiki I think
[16:24:33] <YuviPanda>	 yeah
[16:25:13] <bd808>	 {{done}}
[16:25:19] <bd808>	 woot
[16:25:45] <YuviPanda>	 yay :)
[16:26:10] <bd808>	 I'd forgotten that May made that for us in Zürich
[16:27:17] <valhallasw`cloud>	 YuviPanda: or a duocorn, with two horns forming the V of Vagrant!
[16:27:26] <YuviPanda>	 hahahaha
[16:27:26] <YuviPanda>	 :D
[16:27:48] <valhallasw`cloud>	 http://thatssokeith.files.wordpress.com/2010/07/duocorn.png
[16:27:52] <valhallasw`cloud>	 ;D
[16:32:18] <Merlissimo>	 Coren: you closed https://phabricator.wikimedia.org/T75861 but the dns problem (etc/hosts) still exists
[16:34:12] <wikibugs>	 3Labs-Team, Wikimedia-Labs-Infrastructure, Wikimedia-Labs-tools: maiwiki: s3.labsdb server alias maiwiki.labsdb missing in /etc/hosts and wrong name value in meta__p - https://phabricator.wikimedia.org/T75861#784798 (10Merl) 5Resolved>3Open Ok, but the dns problem still exists:  ``` tools.merlbot@tools-dev:~...
[16:36:57] <Coren>	 Merlissimo: Oh!  Sorry, I hadn't noticed that part of the bug.  The new hosts table has been generated from the new wikis, but I haven't pushed it yet.
[16:37:29] <Coren>	 That said, technically, you're supposed to connect to 'slice' when you want to reach that db (but yes, the maiwiki alias should still be available)
[16:38:04] <valhallasw`cloud>	 err, except that there are not different slices anymore, right?
[16:38:07] <YuviPanda>	 Coren: why are they in hosts files? why can't we just add DNS entries?
[16:38:08] <Merlissimo>	 i am running array jobs each queries one wiki db 
[16:38:20] <valhallasw`cloud>	 not on labs, at least
[16:38:22] <Coren>	 valhallasw`cloud: They remain valid names are are used to spread the load a bit.
[16:38:37] <Merlissimo>	 and input in wiki name that's why i am using this aliases 
[16:39:06] <Coren>	 Merlissimo: Give me 10 m inutes to push the new hosts file.
[16:39:35] <Coren>	 YuviPanda: There's a long story behind that; I'll be happy to go into detail when I'm a bit less busy.  :-)
[16:39:38] <Merlissimo>	 next run of these jobs is at 7 am 
[16:39:44] <YuviPanda>	 Coren: alright :)
[16:39:46] <Merlissimo>	 utc
[16:53:25] <wikibugs>	 3Labs-Team, Wikimedia-Labs-Infrastructure, Wikimedia-Labs-tools: maiwiki: s3.labsdb server alias maiwiki.labsdb missing in /etc/hosts - https://phabricator.wikimedia.org/T75861#784839 (10coren)
[16:53:45] <wikibugs>	 3Labs-Team, Wikimedia-Labs-Infrastructure, Wikimedia-Labs-tools: maiwiki: s3.labsdb server alias maiwiki.labsdb missing in /etc/hosts - https://phabricator.wikimedia.org/T75861#783966 (10coren)
[16:58:28] <Merlissimo>	 Coren: can you clean the error state of three webgrid sge queues?
[17:00:05] <Merlissimo>	 or yuvipanda
[17:00:11] <YuviPanda>	 oh?
[17:00:30] <YuviPanda>	 hmm, so I've never actually touched SGE other than superficially. if Coren isn't around in, say, 10mins, I'll dig?
[17:00:41] <YuviPanda>	 sorry, Merlissimo
[17:01:02] <Merlissimo>	 qmod -cq 'webgrid-lighttpd@*'
[17:10:33] <YuviPanda>	 Merlissimo: alright, let me take a look
[17:11:59] <Merlissimo>	 there is nothing you can do wrong by executing the command i posted above
[17:13:02] <YuviPanda>	 Merlissimo: done
[17:13:13] <YuviPanda>	 well, copy pasting commands from IRC without investigating... :
[17:13:13] <YuviPanda>	 )
[17:13:24] <Merlissimo>	 thx YuviPanda
[17:14:25] <planemad>	 Greetings! I'm trying to load some shapefiles into a postgres database on toollabs
[17:14:54] * YuviPanda waves at planemad
[17:15:39] * planemad waves back
[17:16:44] <planemad>	 psql gives this error: psql: could not connect to server: No such file or directory
[17:17:09] <YuviPanda>	 what is the psql commandline you used?
[17:17:10] <planemad>	 how do I connect to the database?
[17:17:33] <planemad>	 YuviPanda, just 'psql'
[17:17:34] <YuviPanda>	 planemad: https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Database#Connecting_to_OSM_via_the_official_CLI_PostgreSQL documentation :)
[17:17:38] <YuviPanda>	 yeah, that won't work
[17:18:03] <YuviPanda>	 you need to specify which host you want to connect to with -h
[17:18:08] <YuviPanda>	 in this case labsdb1004.eqiad.wmnet
[17:18:54] <planemad>	 my credentials are for pgsql.eqiad.wmnet
[17:19:00] <YuviPanda>	 that would work too
[17:19:08] <YuviPanda>	 I think they're aliases of each other
[17:19:08] <YuviPanda>	 so just use 
[17:19:13] <YuviPanda>	 psql -h pgsql.eqiad.wmnet
[17:19:17] <YuviPanda>	 along with your credentials
[17:21:02] <planemad>	 aha. since psql turned up an error, I thought something was wrong. will try with all the option
[17:21:45] <YuviPanda>	 :)
[17:21:51] <YuviPanda>	 by default it tries to connect to same machine
[17:24:42] <bd808>	 YuviPanda: The issue where I'm still logged into wikitech by have lost my session that allows me to make Nova changes is happening like every day now. :(
[17:24:52] <YuviPanda>	 yup...
[17:24:52] <bd808>	 *but have lost
[17:24:57] <YuviPanda>	 andrewbogott: ^
[17:25:31] <andrewbogott>	 Yeah, for me too.  Probably my attempt to fix it made it worse :(
[17:25:36] <andrewbogott>	 I will look
[17:25:56] <bd808>	 thanks folks
[17:26:22] <wikibugs>	 3Wikimedia-Labs-wikitech-interface: Install Extension:OAuth on wikitech - https://phabricator.wikimedia.org/T70305#784877 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Was done a while ago.
[17:27:18] <Krinkle>	 YuviPanda: pong, re monitoring
[17:27:38] <YuviPanda>	 Krinkle: heya! so it's all gone from icinga now, and you should get monitoring emails from shinken now.
[17:27:51] <YuviPanda>	 Krinkle: can also add cvn to if, if you want
[17:27:55] <bd808>	 YuviPanda: The next thing to fix with labs puppetmaster automation is the same fetch for the labs/private repo
[17:28:16] <YuviPanda>	 bd808: oh, right. don't we already have a cron for that?
[17:28:20] <YuviPanda>	 we could wrap that git pull with this instead
[17:28:23] <bd808>	 nope
[17:28:35] <YuviPanda>	 wait, labs/private isn't kept up to date at all?
[17:28:57] <bd808>	 Not on the self-hosted boxes
[17:29:18] <YuviPanda>	 right.
[17:29:35] <bd808>	 `GIT_SSH=/var/lib/git/ssh git pull --rebase`
[17:29:44] * bd808 has to look that up every time
[17:29:52] <Krinkle>	 YuviPanda: Where is all this located?
[17:29:55] <YuviPanda>	 well, we could just make the current script take a param for prod / labs
[17:30:01] <YuviPanda>	 Krinkle: puppet, shinken module
[17:30:13] <YuviPanda>	 Krinkle: adding a new project involves editing shinkengen.yaml file, and adding contactgroup with same name as project.
[17:30:27] <YuviPanda>	 bd808: file a bug? I'll take a look soon
[17:30:43] <bd808>	 YuviPanda: It's a little weirder than that because of the need to use the ssh wrapper
[17:31:04] <YuviPanda>	 I'm sure we can put that into the script :)
[17:31:19] <YuviPanda>	 sync ops/puppet, sync labs/private, and have it switch on that param
[17:33:47] <YuviPanda>	 Krinkle: example commit: https://gerrit.wikimedia.org/r/#/c/174763/
[17:34:12] <wikibugs>	 3Wikimedia-Labs-tools: Web services continually restarting - https://phabricator.wikimedia.org/T71934#784888 (10coren)
[17:39:13] <bd808>	 YuviPanda: For some day when you get bored -- https://phabricator.wikimedia.org/T75904
[17:39:26] <wikibugs>	 3Labs-Team, Wikimedia-Labs-Infrastructure, Wikimedia-Labs-tools: maiwiki: s3.labsdb server alias maiwiki.labsdb missing in /etc/hosts - https://phabricator.wikimedia.org/T75861#784899 (10coren)
[17:39:41] <wikibugs>	 3Labs-Team, Wikimedia-Labs-Infrastructure, Wikimedia-Labs-tools: maiwiki: s3.labsdb server alias maiwiki.labsdb missing in /etc/hosts - https://phabricator.wikimedia.org/T75861#784901 (10coren)
[17:40:08] <Merlissimo>	 thx Coren
[17:40:58] <Coren>	 Merlissimo: Annoyingly enough, that can't quite be automated for new projects - the place where the data lives is not accessible from Labs so I need to construct and merge the hosts file by hand.
[17:42:39] <Merlissimo>	 Coren: but you can access meta_p which contained this data
[17:43:11] <Coren>	 Hm, that's actually a good point.  I wonder why I never considered it.  :-)
[17:43:39] <Coren>	 I guess I just thought of the meta tables as "output" of the process.
[18:06:04] <gwicke>	 hey
[18:06:41] <gwicke>	 I created a guide for the steps needed to set up a new labs node (geared towards betalabs) at https://wikitech.wikimedia.org/wiki/User:GWicke/betalabs_node_setup
[18:07:53] <gwicke>	 afaik there isn't any step-by-step help for this yet
[18:08:01] <YuviPanda>	 gwicke: cool! :) do you mind emailing the labs-l list?
[18:08:32] <gwicke>	 I could, but before I do so I'd like to ask you to check it for accuracy
[18:09:03] <YuviPanda>	 gwicke: hmm, I haven't ever created one. bd808 perhaps.
[18:09:42] <gwicke>	 I took these notes while figuring out the process, and then followed them for the last node
[18:10:25] <gwicke>	 so I'm pretty certain it works for me, but I'm sure there are bits that could be less confusing or more general
[18:10:35] <YuviPanda>	 gwicke: seems pretty accurate to me.
[18:10:47] <YuviPanda>	 is very deployment-prep specific of course
[18:11:18] <andrewbogott>	 Probably not appropriate for the labs list if it's only about beta
[18:11:26] <gwicke>	 yeah, but I figured it's better to provide one concrete example
[18:11:40] <gwicke>	 the main thing that would change for other projects is the host names
[18:11:41] <YuviPanda>	 andrewbogott: I think it also talks about salt masters and per-project puppetmasters, so useful I think
[18:11:51] <andrewbogott>	 ok
[18:13:02] <gwicke>	 moved it to https://wikitech.wikimedia.org/wiki/Labs_node_setup
[18:13:20] <YuviPanda>	 gwicke: bd808 is here!
[18:21:55] <gwicke>	 mail sent
[18:23:22] <YuviPanda>	 gwicke: ty!
[18:28:51] <gwicke>	 YuviPanda: yw!
[18:31:20] <Superyetkin>	 "table does not exist"?
[18:31:41] <Superyetkin>	 my tools at http://tools.wmflabs.org/superyetkin/index.html do not seem to work
[18:34:29] <Superyetkin>	 hello?
[18:34:51] <YuviPanda>	 hey Superyetkin
[18:35:04] <Superyetkin>	 hello yuvi
[18:35:05] <YuviPanda>	 which table are you getting that error for?
[18:35:14] <Superyetkin>	 trwiki_p.page
[18:35:32] <Superyetkin>	 this has worked before
[18:35:41] <Superyetkin>	 today, there appears to be a problem
[18:36:04] <YuviPanda>	 hmm, looking
[18:36:48] <YuviPanda>	 ok, trwiki is empty...
[18:37:03] <Superyetkin>	 hmm, it broke down?
[18:37:16] <YuviPanda>	 Superyetkin: so, one shard seems to have problem.
[18:37:40] <Superyetkin>	 yeah
[18:37:51] <YuviPanda>	 Superyetkin: in the meantime, you can connect to the 'enwiki.labsdb' host and just issu a 'use trwiki_p' command first.
[18:37:55] <YuviPanda>	 Superyetkin: that shard seems to work fine
[18:38:20] <Superyetkin>	 how long would it take to get back to normal?
[18:38:30] <YuviPanda>	 Coren: ^
[18:38:48] <YuviPanda>	 Coren: c3 doesn't seem to have a copy of trwiki anymore
[18:38:57] <YuviPanda>	 Superyetkin: unsure. but using the hack I mentioned should work for the foreseeable future
[18:38:59] <Coren>	 That's... wait what?
[18:39:11] * Coren boggles a bit.
[18:39:31] * Coren is on it.
[18:40:56] <Superyetkin>	 will there be a maintenance task or something else? is this a replication issue?
[18:41:10] <YuviPanda>	 Superyetkin: unsure yet, Core.n is looking :)
[18:41:24] <Superyetkin>	 ok, thanks
[18:43:26] <Coren>	 Oh odd.  There's an entire slice missing from c3
[18:53:54] <Coren>	 That should be easy to workaround while Sean takes a look.
[19:04:02] <Coren>	 Superyetkin: That should be fixed now.
[19:10:53] <Superyetkin>	 failed again
[19:10:55] <Superyetkin>	 see http://tools.wmflabs.org/superyetkin/olmus_insanlar.php
[19:11:20] <YuviPanda>	 Coren: ^
[19:13:10] <wikibugs>	 3Wikimedia-Labs-Infrastructure, Beta-Cluster: beta: Get SSL certificates for *.{projects}.beta.wmflabs.org - https://phabricator.wikimedia.org/T50501#785345 (10greg)
[19:17:11] <Superyetkin>	 connecting to enwiki.labsdb and running "use trwiki_p" does not work either...
[19:20:51] <YuviPanda>	 Superyetkin: I just tried that, and it seems to work?
[19:21:13] <Coren>	 Superyetkin: Err, that's my fault.  I reshuffled the dbs and managed to actually not fix trwiki which is the one I started this for.  But yes, with enwiki it should work in the meantime.
[19:22:03] <Coren>	 (trwiki was, btw, fixed)
[19:22:06] <Coren>	 (Just now, I mean)
[19:22:49] <Superyetkin>	 thanks Coren, that works
[19:22:50] <Superyetkin>	 http://tools.wmflabs.org/superyetkin/olmus_insanlar.php
[19:23:17] <YuviPanda>	 Coren: are other wikis affected too?
[19:23:48] <Coren>	 YuviPanda: There could still be a couple, but I *think* that I tracked down all the missing ones and redirected them to c2
[19:24:03] <YuviPanda>	 ah, ok
[19:24:29] <Coren>	 It looks like c3 is - for some reason - only getting copies of 6 of the 7 slices.  Not sure why.
[19:25:02] <Coren>	 I'm still looking into it but, in the meantime, I've just switched which server those point to.
[19:25:20] <YuviPanda>	 ok
[19:28:13] <Merlissimo>	 https://phabricator.wikimedia.org/T75493 is a the bug reported for the problem just discussed
[19:42:09] <wikibugs>	 3Wikimedia-Labs-tools: Migrate Tools access request process to Phabricator - https://phabricator.wikimedia.org/T72625#785419 (10yuvipanda)
[19:57:36] <wikibugs>	 3Wikimedia-Labs-Infrastructure, Beta-Cluster: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#785524 (10greg)
[20:19:46] <wikibugs>	 3Wikimedia-Labs-Infrastructure, Beta-Cluster: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#785722 (10greg) p:5Triage>3Normal
[20:19:53] <wikibugs>	 3Wikimedia-Labs-Infrastructure, Beta-Cluster: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#785524 (10greg)
[20:28:52] <wikibugs>	 3Wikimedia-Labs-Infrastructure, Beta-Cluster: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - https://phabricator.wikimedia.org/T71601#785780 (10greg) p:5High>3Normal
[20:37:15] <wikibugs>	 3Wikimedia-Labs-Infrastructure, Beta-Cluster: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#785800 (10bd808) > make new labs project, restrict access a lot  The only downside I see for this plan is that it becomes "yet another way that beta is not lik...
[20:53:05] <wikibugs>	 3Wikimedia-Labs-Infrastructure, Beta-Cluster: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#785867 (10yuvipanda) Can't actually use dynamicproxy, since the cert there is just for *.wmflabs.org. Also, this terminator could be much simpler, since all it...
[21:13:57] <wikibugs>	 3Wikimedia-Labs-Infrastructure, Beta-Cluster: beta: Get SSL certificates for *.{projects}.beta.wmflabs.org - https://phabricator.wikimedia.org/T50501#786000 (10Qgil)
[21:30:56] <wikibugs>	 3Wikimedia-Labs-Infrastructure, Beta-Cluster: beta: Get SSL certificates for *.{projects}.beta.wmflabs.org - https://phabricator.wikimedia.org/T50501#786050 (10greg)
[21:58:37] <LabsRocks>	 is there away to identify blocked users from the database side? I was looking at user_groups table but didn't really have group call block but found 'ipblock-exempt' also don't know what that really means. 
[21:59:52] <legoktm>	 LabsRocks: blocks are stored in the ipblocks table
[22:00:00] <legoktm>	 https://www.mediawiki.org/wiki/Manual:Ipblocks_table
[22:00:06] <legoktm>	 > The name "ipblocks" is a legacy name, as now user account blocks are also stored in this table.
[22:01:42] <legoktm>	 Coren: could you approve https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Smalyshev please? he's a mw core team member :)
[22:03:07] <andrewbogott>	 legoktm: done
[22:03:12] <legoktm>	 thanks!
[22:17:02] <bd808>	 YuviPanda: I was trying to use quarry and I got a 502. Then I tried again and it worked. Is there some potential flakiness with the proxy setup there?
[22:20:31] <mutante>	 deleted the 'autofs' related stuff, ldap::client::autofs , should not have an effect
[22:22:04] <LabsRocks>	 <legoktm> Thanks for the suggestion ipblocks table is helpful. 
[22:44:36] <wikibugs>	 3Wikimedia-Labs-Infrastructure, Beta-Cluster: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#786292 (10hashar) I loose track on the rather long {T50501} , but I commented about using a certificate authority to generate  free certs for beta then inject...
[22:48:16] <wikibugs>	 3Wikimedia-Labs-Infrastructure, Beta-Cluster: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#786302 (10greg) >>! In T75919#786292, @hashar wrote: > I loose track on the rather long {T50501} , but I commented about using a certificate authority to gener...
[23:04:54] <wikibugs>	 3Wikimedia-Labs-wikitech-interface: Stop using an older release of SMW and get back to tracking their master branch - https://phabricator.wikimedia.org/T75940 (10greg) 3NEW p:3Normal
[23:18:47] <grrrit-wm>	 (03PS1) 10Legoktm: Ignore column change-only changes [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/175890 
[23:19:16] <legoktm>	 bd808: ^ just in time!
[23:19:28] <wikibugs>	 3Wikimedia-Labs-wikistats: replace/rewrite largest_html.php / re-add output in other formats for "largest" all-in-one stats table - https://phabricator.wikimedia.org/T38290#786470 (10Dzahn)
[23:19:47] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032 V: 032] Ignore column change-only changes [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/175890 (owner: 10Legoktm)
[23:19:56] <bd808>	 legoktm: \o/
[23:21:30] <legoktm>	 !log tools.wikibugs restarted wikibugs.py listener for https://gerrit.wikimedia.org/r/175890
[23:21:33] <labs-morebots>	 Logged the message, Master
[23:24:33] * bd808 will now test the shushed bot
[23:27:08] <wikibugs>	 3Wikimedia-Labs-wikistats: replace/rewrite largest_html.php / re-add output in other formats for "largest" all-in-one stats table - https://phabricator.wikimedia.org/T38290#786522 (10Dzahn)
[23:31:58] <halfak>	 Hey folks.  Can someone direct me to a good reference for the labsDB schema?
[23:32:11] * halfak is putting together some newbie docs
[23:39:34] <Coren>	 halfak: That depends what you mean by labsdb.  the replicas are (meant to) be copies of production, and except for the meta_p db there are no additions.
[23:39:52] <halfak>	 revision_userindex
[23:39:56] <halfak>	 The blanked fields in user
[23:39:58] <halfak>	 etc. 
[23:40:16] <halfak>	 I was hoping for a summary of differences or something like that. 
[23:40:32] <halfak>	 It's hard for a newcomer to expect there to be a field and have it missing or NULLed. 
[23:41:15] <Coren>	 Ah, I see what you mean.
[23:41:25] <halfak>	 But if there's nothing like that, I'll make due and put creating such a thing on the backlog :) 
[23:41:42] <Coren>	 Well, there is no specific documentation, but the list of views is available in the maintain-replicas script that creates them though.
[23:43:05] <halfak>	 Gotcha.  Thanks.  I think this might be a SOFIXIT situation.  :)  In the meantime, can you link me to this script so that I can reference it?
[23:43:18] <Coren>	 https://git.wikimedia.org/blob/operations%2Fsoftware/HEAD/maintain-replicas%2Fmaintain-replicas.pl;jsessionid=7v1htegfp4ls1iedjqihbumk6
[23:43:52] <Coren>	 Line 90 down has the views that aren't straight 'select * from'; the format should be quite readable.