[00:30:12] Krenair: hi! sorry to bother u again... I'm getting a weird result from static js resources on the beta cluster... [00:31:21] hi [00:31:23] what's up? [00:31:37] Krenair: hey thanks... compare: http://en.wikipedia.beta.wmflabs.org/static/master/extensions/CentralNotice/resources/subscribing/ext.centralNotice.bannerHistoryLogger.js [00:32:18] Lines 257-258 have the old version of a URL param we're using, "bannerLoggerRate" [00:32:52] However Special:Version does point to the most recent commit on CentralNotice master, as it should: http://en.wikipedia.beta.wmflabs.org/wiki/Special:Version [00:33:22] Here is the same file on the commit pointed to there: https://git.wikimedia.org/blob/mediawiki%2Fextensions%2FCentralNotice.git/933caac38e459cb29af472ec46b1bb38e25829b3/resources%2Fsubscribing%2Fext.centralNotice.bannerHistoryLogger.js#L257 [00:33:42] As you can see, it's different from the static resource being served on the beta cluster [00:34:07] yeah [00:34:14] The main issue is that I'd like to make sure I'm working with the most recent version of the code... [00:34:40] Maybe it's different in debug vs non-debug mode? I saw something like this on prod once [00:36:23] so it's all gone from mediawiki-staging on deployment-bastion [00:36:54] and mediawiki [00:37:52] And all of the apaches [00:38:31] hmmm [00:39:44] And indeed if I load it directly instead of going via varnish, it does not appear [00:40:07] (the string bannerLoggerRate) [00:40:11] So it's a cache thing? [00:40:14] I think so [00:40:20] K [00:40:33] I'm just trying to see which I get outside debug mode... [00:41:10] krenair@deployment-mediawiki03:~$ curl --silent -H "Host: en.wikipedia.beta.wmflabs.org" localhost/static/master/extensions/CentralNotice/resources/subscribing/ext.centralNotice.bannerHistoryLogger.js | grep bannerLoggerRate [00:41:10] krenair@deployment-mediawiki03:~$ curl --silent en.wikipedia.beta.wmflabs.org/static/master/extensions/CentralNotice/resources/subscribing/ext.centralNotice.bannerHistoryLogger.js | grep bannerLoggerRate [00:41:11] // URL param bannerLoggerRate can override rate, for debugging [00:41:11] var rateParam = mw.util.getParamValue( 'bannerLoggerRate' ), [00:41:12] krenair@deployment-mediawiki03:~$ [00:42:06] I guess you could SquidUpdate::purge it [00:42:12] But I don't know what else it may be caching wrongly [00:43:16] And purging just that static url won't change how your script runs normally [00:43:24] Any ideas Krinkle_? [00:43:26] or bblack [00:44:19] Krenair: yeah without debug it's serving the new code [00:45:35] Krenair: http://en.wikipedia.beta.wmflabs.org/w/load.php?debug=false&lang=en&modules=ext.centralNotice.bannerHistoryLogger&skin=vector&version=d4342ee94f78&* [00:45:44] ^ serves with the new parameter [00:46:28] without debug [00:46:30] That's good enuf for my testing purposes at hand :) [00:47:16] Krenair: I'm pretty sure this is the Phab task for the same issue on production: https://phabricator.wikimedia.org/T99096 [00:47:34] Heh so in fact if the beta cluster is doing the same as prod, it's doing it right! ;) [00:48:48] Yeah, "within the lifetime of a WMF release branch" is irrelevant here because beta only ever runs the master branch [00:49:44] Ah hmmm [00:50:06] So that does seem like the right task [00:50:16] Is this blocking you from doing anything in particular? [00:51:59] Krenair: no! Like I said, just wanted to be sure that I was getting the new code... Seeing now that it's the case in non-debug mode, it's fine :) [00:52:17] Krenair: thanks so much and sorry 4 the bother!! :D [00:52:39] I don't want to just purge this single page from varnish due to potential inconsistencies with other outdated cached files [00:52:41] np [00:52:48] right... [00:53:20] Krenair: yeah totally np :) [01:29:13] 6Labs, 10wikitech.wikimedia.org: Can't list instances on Special:NovaInstance - https://phabricator.wikimedia.org/T110629#1582693 (10scfc) 3NEW [01:33:18] 6Labs, 10wikitech.wikimedia.org: Can't list instances on Special:NovaInstance - https://phabricator.wikimedia.org/T110629#1582707 (10Krenair) WFM [01:40:42] 6Labs, 10wikitech.wikimedia.org: Can't list instances on Special:NovaInstance - https://phabricator.wikimedia.org/T110629#1582717 (10scfc) After promoting "Tim Landscheidt (Test)" to a projectadmin in Toolsbeta, that user can see the instance list for Toolsbeta. The user "Tim Landscheidt" still sees no instan... [01:41:45] 6Labs, 10wikitech.wikimedia.org: Can't list instances on Special:NovaInstance - https://phabricator.wikimedia.org/T110629#1582718 (10Krenair) Huh. Did you do this in separate browsers or incognito mode or something? If so, tried clearing your cookies? [01:50:25] 6Labs, 10wikitech.wikimedia.org: Can't list instances on Special:NovaInstance - https://phabricator.wikimedia.org/T110629#1582732 (10scfc) I did log in as "Tim Landscheidt (Test)" in an incognito window (Firefox; "private window"?). For "Tim Landscheidt", I just now deleted all cookies for `wikitech.wikimedia... [04:20:33] 6Labs, 10CirrusSearch, 6Discovery, 10wikitech.wikimedia.org, 7Wikimedia-log-errors: Wikitech CirrusSearch jobs throwing exceptions on silver - https://phabricator.wikimedia.org/T110635#1582821 (10Krenair) 3NEW [04:21:55] 6Labs, 10Labs-Infrastructure: Database upgrade MariaDB 10: Metadata access in INFORMATION_SCHEMA causes complete blocks - https://phabricator.wikimedia.org/T71182#1582841 (10Springle) [04:22:13] 6Labs, 10Labs-Infrastructure, 7Database: Database upgrade MariaDB 10: Discrepancies with logging table on different wikis - https://phabricator.wikimedia.org/T71127#1582844 (10Springle) [04:22:52] 6Labs, 10Labs-Infrastructure, 7Database: Database upgrade MariaDB 10: Metadata access in INFORMATION_SCHEMA causes complete blocks - https://phabricator.wikimedia.org/T71182#1582846 (10Krenair) [04:23:06] 6Labs, 10Labs-Infrastructure: Database upgrade MariaDB 10: 600 seconds timeout - https://phabricator.wikimedia.org/T71110#1582848 (10Springle) 5Open>3Resolved [04:23:44] 6Labs, 10Labs-Infrastructure: Database upgrade MariaDB 10: Engine / Option mismatch on table `user_properties` - https://phabricator.wikimedia.org/T70942#1582858 (10Springle) [04:27:36] 6Labs, 10wikitech.wikimedia.org: Keystone tokens truncated when wikitech stores them - https://phabricator.wikimedia.org/T92014#1582866 (10Krenair) a:5Springle>3Andrew [04:29:34] (03PS1) 10Niharika29: Send Community-Tech traffic to #wikimedia-commtech [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/234464 [05:03:14] 6Labs, 10Labs-Infrastructure, 7Database: Database upgrade MariaDB 10: Engine / Option mismatch on table `user_properties` - https://phabricator.wikimedia.org/T70942#1582908 (10Krenair) [05:12:56] (03CR) 10Legoktm: [C: 032] Send Community-Tech traffic to #wikimedia-commtech [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/234464 (owner: 10Niharika29) [05:13:10] (03Merged) 10jenkins-bot: Send Community-Tech traffic to #wikimedia-commtech [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/234464 (owner: 10Niharika29) [05:14:01] !log tools.wikibugs Updated channels.yaml to: 30a3e422993291ab995a487ae1d4bbc2e7cd4013 Send Community-Tech traffic to #wikimedia-commtech [05:14:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL, Master [08:06:14] 6Labs, 10Tool-Labs, 5Patch-For-Review: puppet gridengine-exec start always triggered - https://phabricator.wikimedia.org/T110532#1583123 (10valhallasw) 5Open>3Resolved Seems to work on tools-webgrid-lighttpd-1401 as well! :-) [09:25:31] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Strainu was created, changed by Strainu link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Strainu edit summary: Created page with "{{Tools Access Request |Justification=Create database reports for ro.wp, potentially also for running automatic bots once they've been proven on my computer |Completed=false |..." [10:58:06] (03PS1) 10Jean-Frédéric: More useful output with source for no primkey error [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/234494 [10:59:37] (03CR) 10Jean-Frédéric: [C: 032 V: 032] More useful output with source for no primkey error [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/234494 (owner: 10Jean-Frédéric) [11:05:07] Krenair: checking [11:06:34] Krenair: AndyRussG: In debug mode in beta labs, files are always out of date because the url has generic "master" in it, which means they don't roll over naturally. [11:06:46] Don't use debug mode. [11:07:03] (or don't use beta labs) [11:07:52] Filed as https://phabricator.wikimedia.org/T99096 - known infrastructure issue [13:53:15] 6Labs, 10Tool-Labs, 10Continuous-Integration-Infrastructure: Don't know what to put in setup.py/requirements.txt to satisfy both dpkg-buildpackage and tox-flake8 - https://phabricator.wikimedia.org/T110445#1584057 (10hashar) The `mysql-connector-python` module is [[ http://dev.mysql.com/downloads/connector/p... [13:53:44] 6Labs, 10Tool-Labs, 10Continuous-Integration-Config: Don't know what to put in setup.py/requirements.txt to satisfy both dpkg-buildpackage and tox-flake8 - https://phabricator.wikimedia.org/T110445#1584058 (10hashar) [14:51:57] valhallasw`cloud: still here? ;) [14:52:05] addshore: NO. NEVER. [14:52:06] :> [14:52:11] ;) [14:52:29] so we have a webservice on toools ( tool named wlm-de-utils) and its running out of memery [14:52:48] we looked at the webservice script and found out it read memeory limits from some files in .system/config or something bla bla [14:52:53] can we haz more memory? ;) [14:53:11] eh, I think so, but how does one run out of memory? :| [14:53:21] you've got 4G [14:53:24] php script running a python script mid way through [14:53:50] give me a sec, I'll see what the error was [14:54:15] 1835008 bytes [14:54:49] hmm, so it always has 4GB, thats interesting..... guess we have to dig a little deeper... [14:54:53] eh. what's the issue you saw? did the script just die or did SGE kill it? [14:54:54] well [14:55:00] SGE will kill it if you use > 4GB [14:55:07] but there isn't always 4G available [14:55:11] php alooc memoery issue [14:55:14] because we overcommit the hosts [14:55:20] yeah, that's due to overcommitting [14:56:13] eeehm [14:56:17] Hmmmm [14:56:52] so, tools-webgrid-lighttpd-1411.tools.eqiad.wmflabs has ~400M free [14:57:23] which doesn't really explain why it dies at only 1.8MB :/ [14:57:57] On 1405 [14:58:11] Oh or maybe not... [14:58:31] Can you see the job? [14:58:38] yeah, qstat said 1411 [14:59:09] sorry, I don't really have a solution [14:59:15] What is the PHP men limit? :p [14:59:29] Not different on trusty is it? Something silly like 1mb? [14:59:32] could be [15:00:17] memory_limit = 128M [15:00:23] Mwhhwhmmhmmm [15:01:42] Confused, its not even during at the python bit. [15:02:20] yeah, I'm confused as well. If memory_limit=128M, you should never be in trouble even if only 400M is free [15:03:01] right, digging still [16:19:47] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Strainu was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=175414 edit summary: [16:25:23] valhallasw`cloud: Can you give https://gerrit.wikimedia.org/r/#/c/232285/ a once-over so I can merge before we forget all about how we did that? [16:25:39] ooh, yes, sorry for not looking at it earlier [16:26:19] YuviPanda: you around? [16:27:01] gwicke: sup [16:27:05] hey! [16:27:45] I just wanted to upload a new parsoid deb, but ran into an issue on parsoid-spof; some files are missing since the move from nfs to local storage [16:28:23] one is a config that I reconstructed already, but the other bit is the private gpg key for the repo, which is a bit harder to reconstruct [16:28:52] do you think we still have backups from before the switch? [16:30:26] specifically, I'm looking for /home/gwicke/.gnupg [16:33:24] gwicke: yeah we have the files [16:33:42] gwicke: can this wait till Monday (its 6pm on Friday here :D) or do you need it now? [16:34:31] YuviPanda: it's not that urgent; Monday is fine [16:34:41] thanks! [16:34:56] gwicke: cool :) wanna create a ticket with files you want so I can do that async? [16:35:05] YuviPanda: yes, will do [16:35:11] Hi, I'm trying to setup a crontab for one of the tools on labs. The crontab -e was automatically modified to look like "30 16 * * * /usr/bin/jsub -N cron-tools.database-reports-1 -once -quiet python main.py" Does that look right? It didn't run. Am I missing some step? I was reading [16:35:11] https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Grid#Scheduling_jobs_at_regular_intervals_with_cron [16:35:45] Niharika: try specifying the full path to the file? [16:35:58] YuviPanda: That's the full path. [16:36:06] Main.py? [16:36:19] Wait, full path with respect to home dir? [16:36:27] Let me try that. [16:36:30] Full path would be /data/project/toolname/something/main.py [16:36:35] So full full path [16:37:07] Ahh. [16:37:13] Niharika: also, there should be something in ~/cron-tools.database-reports-1.err [16:37:56] valhallasw`cloud: Yep, file not found. Path issue then. Thanks. :) [16:38:51] 6Labs: Restore some files from /home/gwicke - https://phabricator.wikimedia.org/T110698#1584533 (10GWicke) 3NEW a:3yuvipanda [16:39:14] Thanks gwicke [16:40:40] YuviPanda: thanks as well! [16:42:38] having a prod debian repo that devs can push to would be even better, but that's a more complicated project [16:44:00] gwicke: you should check out wikitech.Wikimedia.org/wiki/Aptly [16:44:23] yeah, just saw that you packaged it [16:44:35] we've been using it for a couple years now [16:45:26] I pushed for setting up a central aptly repo a while ago, so that we can publish official service debs for third-party devs from a central place [16:49:20] https://www.mediawiki.org/wiki/Packaging has some old info on that [16:50:40] YuviPanda: what's the advantage of this over an nfs-shared dir? [16:51:39] valhallasw`cloud: no NFS? :) [16:51:57] basically using NFS requires that you have NFS, and specifically /data/project enabled on all your instances [16:51:58] this does not [16:52:03] mmm. [16:52:06] so that's a big one, IMO. [16:52:24] but now you are dependent on a random extra server [16:52:40] you can reuse servers for this. [16:53:47] the question that we don't have an answer to (yet!) is 'how to back this up?' [16:53:51] current answer is: NFS [16:54:01] but, you can get away with having NFS on only one host [16:54:08] hopefully within end of the year we'll have a better bacup story [16:54:10] *backup [16:56:29] mmm, ok [16:56:38] valhallasw`cloud: "please check if continuous jobs died in this process; if this happens” you mean, if they didn’t die, or if they did die? [16:56:58] 'if jobs did die, try another process' [16:57:11] because we didn't actually test this process [19:34:34] 6Labs, 10Tool-Labs, 10Labs-Infrastructure, 7Regression: Tools Labs proxy should not overwrite error page - https://phabricator.wikimedia.org/T103662#1585277 (10valhallasw) I've been digging a bit into the nginx lua module, and I think we can actually do something really cool: wrap the actual error page wit... [19:35:41] 6Labs, 10Tool-Labs, 10Labs-Infrastructure, 7Regression: Tools Labs proxy should not overwrite error page - https://phabricator.wikimedia.org/T103662#1585283 (10yuvipanda) Problem with the lighttpd approach is not sure what that does for uwsgi, nodejs, etc. [19:42:48] 6Labs, 10Tool-Labs, 10Labs-Infrastructure, 7Regression: Tools Labs proxy should not overwrite error page - https://phabricator.wikimedia.org/T103662#1585311 (10valhallasw) The disadvantage to that is that fcgi calls are never decorated with error pages in lighttpd, so a ``` YuviPanda: I'm a bit afraid the lua option will kill nginx :P [19:51:41] I have an interesting problem! When I run a SQL query on labs, it takes forever, but when I run it on Quarry it runs in 30 seconds or so [19:51:48] other queries are all fine, just this one query [19:51:56] anybody willing to help [19:51:58] ? [19:53:44] if it's the exact same query, you're probably running it on a different server [19:54:13] the difference in runtime can be caused by load on the server, or by the results being in the cache on the server quarry uses [19:54:38] valhallasw`cloud: that makes sense. But how come all other queries take almost the same amount of time? [19:54:49] Also, it is nothing special, just a simple set of three joins on indexed columns [19:55:37] I guess there could be something wrong in the index definitions, but that seems unlikely [19:55:42] http://quarry.wmflabs.org/query/3759 [19:56:10] 15 seconds [19:56:35] valhallasw`cloud: can't understand what a 15 sec query would just not run there [19:56:44] or not finish* [19:58:22] ...I just suggested explanations? [19:59:15] valhallasw`cloud: thanks. What you described are a set of possibilities that are each unlikely. I can't deny the possibility thoug [19:59:19] it's /also/ really slow on labsdb1002, so I'm going for the 'cache' explanation [19:59:29] how are 'load' and 'cache' unlikely? [19:59:46] load unlikely because all other queries are fast [20:00:01] you can't have a high load only when a spceific query is run [20:00:17] cache is likely, I give you that [20:01:17] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Yassinela1 was created, changed by Yassinela1 link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Yassinela1 edit summary: Created page with "{{Tools Access Request |Justification=What would you like to use your Tool Labs account for? To do some maintenance work in the Arabic Has this request been completed by an a..." [20:06:46] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Yassinela1 was modified, changed by Yassinela1 link https://wikitech.wikimedia.org/w/index.php?diff=175437 edit summary: [20:07:59] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Yassinela1 was modified, changed by Yassinela1 link https://wikitech.wikimedia.org/w/index.php?diff=175439 edit summary: [20:21:01] valhallasw`cloud: I also don't see any indexes for any tables. When I run "SHOW INDEXES FROM revision" I get nothing [20:21:03] is that normal? [20:30:17] yes [20:30:42] that is, they are not shown (they do exist) [20:31:08] but you can't see the indices unless you also have select rights on the underlying table [20:41:38] I'm having some issues getting into the puppet management screens for deployment-prep instances. They are coming up as "the specified resource does not exist." [20:41:44] Example https://wikitech.wikimedia.org/w/index.php?title=Special:NovaInstance&action=configure&project=deployment-prep&instanceid=0c834b6d-0f30-4d68-96ab-3c02386bc2ad®ion=eqiad [20:42:03] known issue or something new? [20:42:47] I'm running private puppetmaster in labs and I started getting this: Error: Could not send report: Connection refused - connect(2) for "wdqs-puppetmaster.wikidata-query.eqiad.wmflabs" port 8140 [20:43:10] looks like some firewall issue maybe but I'm not sure how to fix it [20:44:01] it worked before but apparently something changed :( [20:46:03] bd808: there were some comparable issues yesterday. I forgot what the fix was though -- andrewbogott should know :-) [20:46:35] andrewbogott: Can you help be figure out how to access https://wikitech.wikimedia.org/w/index.php?title=Special:NovaInstance&action=configure&project=deployment-prep&instanceid=0c834b6d-0f30-4d68-96ab-3c02386bc2ad®ion=eqiad ? [20:46:41] that’s using a project-local puppetmaster? I’d say it just needs a restart. [20:46:43] It worked a couple hours ago [20:46:43] SMalyshev: connection refused sounds like the firewall is working but the puppetmaster is offline [20:47:16] valhallasw`cloud: I see master running... [20:47:24] hmm... now I get Warning: Connection timed out - connect(2) for "wdqs-puppetmaster.wikidata-query.eqiad.wmflabs" port 8140 [20:47:39] bd808: I think we had some other issues like that yesterday — it worked to remove a user from the project and replace them. Want me to do that for you? [20:47:48] (That page works for me, btw) [20:47:55] SMalyshev: running doesn’t mean working :) [20:47:56] SMalyshev: try restarting it? [20:48:15] andrewbogott: weird. Sure lets try the in and out of project dance [20:48:29] valhallasw`cloud: restarted, it said OK [20:49:02] but puppets on other hosts don't seem to work [20:49:32] valhallasw`cloud: when I check iptables I don't see port 8140 in the list, is it normal? [20:50:05] bd808: I did a schema update for the keystone db on Wednesday, it seems to have scrambled some transitory things... [20:50:10] try logging out and in again, and see if things are better? [20:50:16] will do [20:51:12] andrewbogott: that worked. I can see the screen now [20:51:20] weird but good [20:54:27] SMalyshev: eh, not sure. I wouldn't expect iptables to be set up at all :/ [20:54:34] but maybe that's a puppetmaster specific thing -- not sure [20:55:49] SMalyshev: iptables isn't setup on toolsbeta-puppetmaster3 [20:56:01] SMalyshev: so the only relevant firewall is the one in openstack [20:56:35] valhallasw`cloud: so I see here: https://wikitech.wikimedia.org/wiki/Nova_Resource:Wdqs-puppetmaster.wikidata-query.eqiad.wmflabs it has only standard roles.... so I wonder then where iptables came from. I just rebooted it and iptables are there [20:57:56] SMalyshev: sorry, I have no idea :( [20:58:55] seems to have ferm enabled [21:01:01] SMalyshev: there may be two levels of filtering… there’s security groups (which you edit via wikitech) and then ferm/iptables/whatever you have installed on your instance. [21:23:15] valhallasw`cloud: We just replaced ldap certs — did you notice any interruption with logins or sudo or anything? And, do things look fine now? [21:23:27] andrewbogott: let me take a look [21:23:36] thans [21:23:38] k [21:24:13] andrewbogott: login and sudo seems to work. ldaplist as well [21:24:46] ok, works for me too. Thanks for checking. [21:53:49] andrewbogott: Labs is finding new and exciting ways to hate me. Now seeing "sudo: unknown uid 3518: who are you?" on deployment-tmh01.deployment-prep.eqiad.wmflabs and then when I logged out it won't let me back in. [21:54:03] I can log into other deployment-prep hosts [21:55:28] and "The specified resource does not exist" is back for https://wikitech.wikimedia.org/w/index.php?title=Special:NovaInstance&action=configure&project=deployment-prep&instanceid=0c834b6d-0f30-4d68-96ab-3c02386bc2ad®ion=eqiad and persisted through logout/login cycle [21:56:43] * bd808 will try removing and re-adding himself [22:00:24] andrewbogott: ok, removing myself from deployment-prep, adding myself back and then logging out and back in got me access to the configure screen again (that bug sucks). [22:00:51] deployment-tmh01.deployment-prep.eqiad.wmflabs is still refusing to let me ssh in however [22:07:25] Restarting nslcd via salt fixed my ssh access to deployment-tmh01 [22:31:18] YuviPanda: If I want to run a really expensive, long running query on Tool Labs (15 minutes+) as a weekly bot job what's the best way to do that, or should I not do it at all? [22:32:36] kaldari: 15mins isn't long running :) so go for it [22:32:50] Quarry's default time limit is 20mins now [22:33:54] OK [22:44:27] is DirectoryIndex allowed to be used in public_html/.htacess? [22:50:39] fhocutt: these are served via lighttpd not apache [22:50:43] So no .htaccess [22:50:59] hah, that would explain it [22:51:02] this is a very old bot [22:51:12] You can put a .lighttpd.conf file in $HOME to override [22:51:17] Must be really really old [22:51:20] it is! [22:51:24] Citation Bot [22:51:28] We switched off apache more than a year and something ago [22:52:25] yeah, most of the config stuff is definitely left over from toolserver [22:52:49] along with the DB it would be maintaining... [22:53:03] Right... [22:53:10] I don't envy your task [22:53:41] ...yeah. Kind of fun to untangle in a "I certainly am learning" way [22:54:43] let's delete that, then. [22:55:35] :) [22:56:16] there's also a "mongo.creds.json" file, is that something that makes Labs go or something mysterious? [22:56:47] bd808: BTW we added support for per host hiera overrides to wikitech [22:57:06] Hiera:$project/host/$hostnamr [22:57:31] In our grand tradition this hadn't been documented anywhere but it got merged only a few days ago [22:57:36] I'll add it to the docs soon [22:57:48] is there a way to combine revision and the archive table so that it can be queried as a single table? [23:01:31] fhocutt: mongo... I thought I had deleted all of them. Was a failed experiment [23:01:45] all right, that goes too! [23:17:19] andrewbogott, YuviPanda: Hey. Any ideas about https://phabricator.wikimedia.org/T110629 ? [23:17:29] It's not affecting me, but I don't know how many users it does affect. [23:17:45] andrewbogott: ^ [23:18:02] (I'm about to crash soon, almost 0130am) [23:18:06] And is causing issues for scfc :( [23:20:04] Krenair, I'm out for the day, but removing a user from a project and readding fixes it. [23:20:27] removing the affected user? [23:20:34] or having the affected user remove someone else? [23:32:47] Affected user.