[03:34:52] PROBLEM - Puppet staleness on tools-exec-15 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [43200.0] [06:35:24] PROBLEM - Puppet failure on tools-exec-12 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:37:33] PROBLEM - Puppet failure on tools-exec-07 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:40:40] PROBLEM - Puppet failure on tools-exec-12 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:02:34] RECOVERY - Puppet failure on tools-exec-07 is OK: OK: Less than 1.00% above the threshold [0.0] [07:05:23] RECOVERY - Puppet failure on tools-exec-12 is OK: OK: Less than 1.00% above the threshold [0.0] [07:09:59] 10Tool-Labs: Get rid of tools.wmflabs.org manually from /etc/hosts - https://phabricator.wikimedia.org/T92234#1103569 (10yuvipanda) 3NEW [09:19:59] RECOVERY - Puppet staleness on tools-exec-15 is OK: OK: Less than 1.00% above the threshold [3600.0] [09:29:23] 10Tool-Labs, 6operations, 7Puppet: cron's puppet-run fails silently if apt-get update fails - https://phabricator.wikimedia.org/T92239#1103738 (10scfc) 3NEW [10:03:55] 10Tool-Labs: Get rid of tools.wmflabs.org manually from /etc/hosts - https://phabricator.wikimedia.org/T92234#1103846 (10scfc) 5Open>3Resolved a:3scfc ``` [tim@passepartout ~]$ pdsh -f 5 -g tools sudo sed -i.bak -e "'/^10\\.68\\.17\\.139 tools-webproxy tools\\.wmflabs\\.org\$/d'" /etc/hosts [tim@passeparto... [10:12:50] 10Tool-Labs: apt-get update fails on tools-exec-15 - https://phabricator.wikimedia.org/T92248#1103873 (10scfc) 3NEW [10:13:35] 10Tool-Labs: apt-get update fails on tools-exec-15 - https://phabricator.wikimedia.org/T92248#1103885 (10scfc) [11:12:57] 6Labs, 10Tool-Labs, 7Tracking: Replace bigbrother and ssh-cron-thingy with service manifests - https://phabricator.wikimedia.org/T90561#1104012 (10scfc) I wrote somewhere else that IIRC I wouldn't want to mash up `bigbrother` with `jstart`, and the same goes for `crontab` :-). In general, I'd prefer if we s... [11:17:18] 6Labs, 10Tool-Labs, 7Tracking: Replace bigbrother and ssh-cron-thingy with service manifests - https://phabricator.wikimedia.org/T90561#1104026 (10yuvipanda) (I'll note that the only thing I really want is to make sure that we have service manifests. How we do that is still up to discussion. Anti-NIHness is... [11:48:35] 10Tool-Labs, 6operations, 7Puppet: cron's puppet-run fails silently if apt-get update fails - https://phabricator.wikimedia.org/T92239#1104073 (10faidon) This is on purpose — we run apt-get update before puppet as to pick up potentially new packages (e.g. in our own apt repository) and properly enforce ensur... [11:52:20] 10Tool-Labs, 6operations, 7Puppet: cron's puppet-run fails silently if apt-get update fails - https://phabricator.wikimedia.org/T92239#1104078 (10yuvipanda) (on labs too, this got picked up as a puppet staleness failure) [12:11:13] andrewbogott_afk: interesting. I can’t uncheck role::deployment::deployment_servers::labs from deployment-bastion page. if I uncheck and save, it just keeps coming back up [12:27:11] 10Tool-Labs, 6operations, 7Puppet: cron's puppet-run fails silently if apt-get update fails - https://phabricator.wikimedia.org/T92239#1104143 (10scfc) I know, that's why I found the error :-). But it took me quite some time to figure out that Puppet is //not// failing, but `apt-get update`. If: ``` timeo... [12:47:13] 10Tool-Labs, 6operations, 7Puppet: cron's puppet-run fails silently if apt-get update fails - https://phabricator.wikimedia.org/T92239#1104178 (10chasemp) I think the idea is that if the puppet initiated apt update fails then puppet is failing indeed. [12:48:59] 10Tool-Labs, 6operations, 7Puppet: cron's puppet-run fails silently if apt-get update fails - https://phabricator.wikimedia.org/T92239#1104192 (10yuvipanda) If I retitle the bug to 'redirect apt-get output to puppet log', would that be an accurate assessment of what you're asking for, @scfc? That sounds usef... [12:50:26] 10Tool-Labs, 6operations, 7Puppet: cron's puppet-run fails silently if apt-get update fails - https://phabricator.wikimedia.org/T92239#1104194 (10faidon) Logging apt-get update's -qq output in the logfile doesn't sound like a bad idea at all! @scfc, do you want to do the honors? :) [12:59:30] 10Tool-Labs, 6operations, 7Puppet: cron's puppet-run fails silently if apt-get update fails - https://phabricator.wikimedia.org/T92239#1104211 (10scfc) Well, in that case perhaps the monitoring should be condensed to "Everything's okay" and "Something's broken", so that I could spend more hours debugging tha... [13:02:44] 10Tool-Labs, 6operations, 7Puppet: cron's puppet-run fails silently if apt-get update fails - https://phabricator.wikimedia.org/T92239#1104230 (10yuvipanda) >>! In T92239#1104211, @scfc wrote: > Well, in that case perhaps the monitoring should be condensed to "Everything's okay" and "Something's broken", so... [13:03:29] 10Tool-Labs: Get rid of tools.wmflabs.org manually from /etc/hosts - https://phabricator.wikimedia.org/T92234#1104232 (10coren) Yes, I fixed this last night. The IP override in /etc/hosts is a necessary wart caused by the fact that the public address is not reachable from within the labs network (a limitation o... [13:05:19] 10Tool-Labs: Get rid of tools.wmflabs.org manually from /etc/hosts - https://phabricator.wikimedia.org/T92234#1104236 (10yuvipanda) >>! In T92234#1104232, @coren wrote: > Yes, I fixed this last night. The IP override in /etc/hosts is a necessary wart caused by the fact that the public address is not reachable f... [13:10:09] hey, is it possible to programmatically create and destroy wikis on Vagrant? [13:14:03] It looks like I should use puppet, but I'm not sure where to put the manifests… [13:15:58] 10Tool-Labs, 6operations, 5Patch-For-Review, 7Puppet: cron's puppet-run fails silently if apt-get update fails - https://phabricator.wikimedia.org/T92239#1104248 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Thank you for reporting :) [13:17:54] 10Tool-Labs: Get rid of tools.wmflabs.org manually from /etc/hosts - https://phabricator.wikimedia.org/T92234#1104251 (10scfc) Or https://gerrit.wikimedia.org/r/#/c/146000/ which bears a +2 from a familiar name :-). [13:45:12] 6Labs, 10Wikidata: wdq.wmflabs.org does not update (data week old) - https://phabricator.wikimedia.org/T89583#1104295 (10JanZerebecki) 5Open>3Resolved [13:45:44] 6Labs, 10Wikidata: wdq.wmflabs.org does not update (data week old) - https://phabricator.wikimedia.org/T89583#1039840 (10JanZerebecki) Seems to work fine now, please reopen if I missed something. [13:52:19] 10Tool-Labs: webservice2 failing to start Python web services with xml.etree.ElementTree.ParseError - https://phabricator.wikimedia.org/T92039#1104306 (10scfc) You're right, `qstat_xml()` does not return valid XML :-). In `labs/toollabs`'s `www/content/status.php`, we worked around that with `sed`. I just test... [14:24:48] Hey, is it possible to have a web proxy with a wildcard? [14:25:17] * werdna looks at YuviPanda hopefully? [14:25:37] werdna: hey! nope, sadly. [14:25:41] hmm [14:25:48] so vagrant has a multiwiki feature [14:26:01] but it requires you to distinguish by domain, not by path [14:26:03] you can create multiple proxies. [14:26:12] just point them to the same host / port [14:26:13] I want to be able to create up to 50, on the fly [14:26:19] ah, right. no. [14:26:49] hmmm [14:27:34] any idea how I could most easily and reproducibly and without-writing-too-much-extra-code-ily have a server that serves tens of wikis which can be created / destroyed on the fly? [14:28:06] not easily and reproducibly no. [14:28:41] ideas? plenty. but none of them are easy, I think :) [14:28:42] could I, for example, open my server to the big bad web and ask somebody really nicely for a wildcard DNS record? [14:29:04] oh, if you just want a wildcard DNS, I think we can give you that [14:29:08] and give you a public IP [14:29:25] so if vagrant does the ‘tens of wikis created / destroyed on the fly’ fine [14:29:27] hmmm, where do I phabricate that? [14:29:29] we can do the IP / DNS stuff [14:29:52] werdna: just put it on the Labs project, along with some justification, perhaps. would probably need hand-editing LDAP. [14:30:17] I really need to get that new laptop [14:30:18] this one has awful wifi [14:30:24] werdna: shouldn’t be hard, though. very doable, just a public IP + wildcard DNS [14:31:03] thanks, I'll Phabricate it (and use my hosts file for testing) [14:31:44] werdna: cool. [14:32:36] ("Phabricate" is totally a verb) [14:34:21] ‘course it is [14:39:49] Phabricate'd: https://phabricator.wikimedia.org/T92273 [14:40:38] werdna: which project is reflex in? [14:40:40] design? [14:41:13] YuviPanda: yes indeed [14:41:36] werdna: alright. you have one public IP now. You can give that to yourself by ‘Manage addresses’ in wikitech sidebar [14:41:46] werdna: but no SSL tho, if you use your own public IP [14:41:52] 6Labs, 10REFLEX: Public IP and Wildcard DNS for REFLEX project - https://phabricator.wikimedia.org/T92273#1104460 (10werdna) [14:42:14] YuviPanda: okay, I can live with that [14:42:34] 6Labs, 10Wikimedia-Hackathon-2015: Organize Wikimedia Labs activities at the Wikimedia Hackathon 2015 - https://phabricator.wikimedia.org/T92274#1104472 (10Qgil) 3NEW [14:42:35] 6Labs, 10REFLEX: Public IP and Wildcard DNS for REFLEX project - https://phabricator.wikimedia.org/T92273#1104479 (10yuvipanda) Increased IP address quota to 1. @andrew Is it possible to do a wildcard domain via wikitech? [14:43:29] YuviPanda: I guess not https://www.dropbox.com/s/u55bjsy24qhoeqh/Screenshot%202015-03-10%2015.43.21.png?dl=0 [14:44:36] 6Labs, 10REFLEX: Public IP and Wildcard DNS for REFLEX project - https://phabricator.wikimedia.org/T92273#1104483 (10werdna) @yuvipanda [I guess not](https://www.dropbox.com/s/u55bjsy24qhoeqh/Screenshot%202015-03-10%2015.43.21.png?dl=0) [14:44:41] hi, so on the login server, why do some people use "tools-webproxy" instead of "tools.wmflabs.org"? starting at about 10:15 GMT today the tools-webproxy doesn't equate to tools.wmflabs.org [14:45:03] you'll get "couldn't resolve host 'tools-webproxy'" [14:45:18] tools-webproxy used to be the old internal name of the proxy [14:45:33] but that shoudl never be considered stable. tools.wmflabs.org is the thing that’s supposed to work.. [14:45:40] ok, good to know [14:45:49] 6Labs, 10Wikimedia-Hackathon-2015: Organize Wikimedia Labs activities at the Wikimedia Hackathon 2015 - https://phabricator.wikimedia.org/T92274#1104489 (10Qgil) Can you nominate a main contact for the Wikimedia Labs area, please? We are doing the same for the other main areas. Also, I wonder whether we could... [14:45:56] thanks [14:51:51] 6Labs, 10Wikimedia-Hackathon-2015: Organize Wikimedia Labs activities at the Wikimedia Hackathon 2015 - https://phabricator.wikimedia.org/T92274#1104510 (10Qgil) [15:21:23] I would appreciate a review of https://gerrit.wikimedia.org/r/#/c/195472/ from anyone familiar with mediawiki and db schemas [15:21:34] For various reasons it’s going to be hard for me to test that patch before running it in prod [15:22:09] legoktm: ^ when you’re around? [15:23:56] YuviPanda: that might (maybe) fix log-out-and-in-again on wikitech [15:24:46] yup! wheeeee [15:36:53] (03PS1) 10Jforrester: Add some obvious repos to -releng (and scap to -operations too) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/195569 [15:36:56] (03CR) 10jenkins-bot: [V: 04-1] Add some obvious repos to -releng (and scap to -operations too) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/195569 (owner: 10Jforrester) [15:38:53] (03PS2) 10Jforrester: Add some obvious repos to -releng (and scap to -operations too) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/195569 [15:57:35] 6Labs, 10hardware-requests, 6operations: Hardware for Designate - https://phabricator.wikimedia.org/T91277#1104754 (10Andrew) [16:10:46] 6Labs, 10hardware-requests, 6operations: Hardware for Designate - https://phabricator.wikimedia.org/T91277#1104814 (10Andrew) Quick conversation with Mark confirms that this box should have a public IP, same as silver or Horizon. We'll figure out about rabbitmq communication later on. [16:17:54] 10Tool-Labs, 6Wikisource: Djvulibre package is lacking into Labs; it's needed to manage djvu files by bots in wikisource projects - https://phabricator.wikimedia.org/T58972#1104954 (10Aklapper) [16:34:25] 6Labs, 10Wikimedia-Hackathon-2015: Organize Wikimedia Labs activities at the Wikimedia Hackathon 2015 - https://phabricator.wikimedia.org/T92274#1105123 (10coren) [16:42:57] 10Tool-Labs: apt-get update fails on tools-exec-15 - https://phabricator.wikimedia.org/T92248#1105141 (10scfc) 5Open>3Resolved a:3scfc After comparing `/var/lib/apt` on `tools-login` and `tools-exec-15`, the major difference was a file in `/var/lib/apt/lists/partial`. So I was bold: ``` scfc@tools-exec-1... [16:45:32] 10Tool-Labs-tools-Other, 6Commons: Move catbot from toolserver to toollabs - https://phabricator.wikimedia.org/T63825#1105179 (10Aklapper) [16:45:34] 10Tool-Labs-tools-Other, 6Commons: Move contests from toolserver to tool labs - https://phabricator.wikimedia.org/T63826#1105181 (10Aklapper) [16:45:36] 10Tool-Labs-tools-Other, 6Commons: Do something with globalusage account on Toolserver - https://phabricator.wikimedia.org/T63827#1105180 (10Aklapper) [16:46:04] 10Wikimedia-Labs-Infrastructure, 6Commons: CatScan doesn't load (times out) - https://phabricator.wikimedia.org/T73336#1105203 (10Aklapper) [17:07:52] (03PS1) 10Yuvipanda: Make mwdeploy key have a passphrase [labs/private] - 10https://gerrit.wikimedia.org/r/195595 [17:08:19] (03CR) 10Yuvipanda: [C: 032 V: 032] Make mwdeploy key have a passphrase [labs/private] - 10https://gerrit.wikimedia.org/r/195595 (owner: 10Yuvipanda) [17:18:53] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10Tool-Labs, 10Tool-Labs-tools-Article-request, and 9 others: Labs' Phabricator tags overhaul - https://phabricator.wikimedia.org/T89270#1105390 (10Aklapper) a:3Aklapper Assigning to me again (temporarily unassigned it to avoid cookie-licking) and I need to... [17:43:19] (03PS1) 10Yuvipanda: Remove mwdeploy keys [labs/private] - 10https://gerrit.wikimedia.org/r/195603 [17:43:34] (03CR) 10Yuvipanda: [C: 032 V: 032] Remove mwdeploy keys [labs/private] - 10https://gerrit.wikimedia.org/r/195603 (owner: 10Yuvipanda) [17:48:17] where's wiki.toolserver.org now? [17:51:25] 10Tool-Labs-tools-Other, 6Commons: Move catbot from toolserver to toollabs - https://phabricator.wikimedia.org/T63825#1105523 (10Multichill) 5Open>3Resolved p:5Triage>3Normal a:3Multichill [17:51:26] 10Tool-Labs-tools-Other, 7Tracking: [tracking] toolserver.org tools that have not been migrated - https://phabricator.wikimedia.org/T60865#1105528 (10Multichill) [17:56:27] Glaisher: coren was handling that. [17:56:31] 10Tool-Labs, 5Patch-For-Review: webservice2 failing to start Python web services with xml.etree.ElementTree.ParseError - https://phabricator.wikimedia.org/T92039#1105576 (10yuvipanda) Seems to work - I tested on a couple of services :) !close [17:56:50] YuviPanda: it's not available anywhere atm? [17:57:03] looks like. [17:57:08] Coren: what happened to wiki.toolserver.org? [17:57:18] 10Tool-Labs, 5Patch-For-Review: webservice2 failing to start Python web services with xml.etree.ElementTree.ParseError - https://phabricator.wikimedia.org/T92039#1102116 (10yuvipanda) a:5yuvipanda>3scfc [17:58:57] YuviPanda: Can you restart the webservice for http://tools.wmflabs.org/catscan2/ ? [17:59:05] alright [17:59:31] multichill: done [17:59:35] it has a .bigbrotherrc file... [17:59:50] multichill: is it working now? [18:00:24] Yeah, seem to be gettin output now :-) Thanks [18:00:52] multichill: yw [18:01:09] Glaisher: do file a bug? [18:01:18] * YuviPanda is super sleepy, it’s a bit late, should probably sleep [18:01:31] YuviPanda: lots of open bugs about it already, I just found [18:01:47] ah, ok. [18:02:06] https://phabricator.wikimedia.org/T62220 blocked bys [18:02:30] hmm, ok [18:02:41] I guess nobody ended up owning that. [18:59:26] legoktm: thanks. Is there a maintenance command I need to run after deploying that change? [18:59:31] 10Wikibugs: wikibugs is having issues joining channels on startup - https://phabricator.wikimedia.org/T92301#1105754 (10Legoktm) 3NEW [19:00:39] andrewbogott: we don't run update.php on the cluster, springle normally does the schema changes across the cluster, but depending on how large the table is, and since it's wikitech you might just be able to do it manually [19:01:45] legoktm: Sean says ‘I've altered openstack_tokens.token to varbinary(2048).’ [19:02:11] Does that mean I don’t need to run update.php? [19:02:27] nope! [19:02:39] ok :) [19:02:42] * bd808 introduces andrewbogott to legoktm [19:02:52] since that extension only runs on the one host… [19:03:02] Is there any more to it than 1) sync 2) run update.php? [19:03:22] I don't think you need to even sync [19:03:26] no update.php on the cluster [19:03:32] there's a php change right? [19:03:38] * andrewbogott stands back [19:03:43] the php change is just adding the update hook [19:03:50] *nod* [19:03:57] if the table has already been altered, new inserts shouldn't be truncated [19:04:02] so if sean made the db change you are basically done [19:04:21] oh? I haven’t tested it :) [19:04:29] the patch is still necessary, right? Just so that the code reflects reality? [19:04:35] yeah [19:04:36] yes [19:04:49] ok then! [19:05:26] Brad has a comment on the patch that the sql syntax may be wrong... [19:05:31] And, let’s see… I need to make a submodule commit. Is there any policy about one patch per submodule commit? [19:06:02] I think you can skip the submodule change and just let the patch ride the train really [19:06:13] ok, that’s easy :) [19:06:14] nobody will be running update.php on the cluster to be confused [19:06:59] But the patch should be amended to fix the sql syntax problem ;) [19:07:38] * legoktm is doing that [19:07:48] legoktm: thanks! [19:09:49] bd808: so is it generally the case that if I +2 a patch in an extension, it just gets deployed normally with the deployment train? I thought that +2 w/out immediate sync was forbidden. [19:10:26] andrewbogott: the +2 & sync is only for mediawiki-config and puppet [19:10:38] ah, great. [19:10:49] core and extensions are expected to drift ahead of deployment [19:12:34] * andrewbogott rewards legoktm by adding many more review tasks [19:12:44] heh [19:12:46] alright :P [19:30:38] !log tools.wikibugs Updated channels.yaml to: 614ee42338f6ab3f8d0705d3f0358523189af00e send WMT bugs to #wmt [19:30:43] Logged the message, Master [20:16:50] eh [20:16:54] what happeend? [20:21:15] hi, grrrit-wm died [20:21:21] but the docs say [20:21:22] Grrrit-wm is managed by Bigbrother, so manual restarts should not be necessary [20:21:34] what is needful [20:22:16] legoktm: how many times do you need to deploy the #wmt change :p [20:22:43] and mutante, legoktm I think also handles grrrit-wm so [20:23:00] JohnFLewis: I was just reloading it, and it happens to spit out the latest git commit :P [20:23:18] mutante: ^ go back to work then :p [20:23:20] 2015-03-03 22:30:32,392 - irc3.wikibugs - DEBUG - > PING 3033084 [20:23:20] 2015-03-03 22:30:32,502 - irc3.wikibugs - DEBUG - PONG ping-pong (3033084) [20:23:20] 2015-03-03 22:31:45,290 - irc3.wikibugs - DEBUG - > QUIT :INT [20:23:21] so helpful [20:24:10] mutante: hmm, someone moved it to ".bigbrotherrc.disabled" [20:24:22] helpfully they did not state why :( [20:24:49] hrmm, gotcha [20:25:07] https://phabricator.wikimedia.org/T92313?workflow=create [20:25:23] thanks legoktm [20:25:31] Your job 8825345 ("lolrrit-wm") has been submitted [20:28:45] Hi grrrit-wm [20:33:21] (03PS1) 10John F. Lewis: create wikisets.php config file [labs/tools/WMT] - 10https://gerrit.wikimedia.org/r/195659 (https://phabricator.wikimedia.org/T91941) [20:42:47] and there it goes again [20:48:44] wikibugs is also gone [20:48:48] again??? [20:48:48] wtf [20:49:23] 2015-03-10T20:41:40.646Z - error: Caught error in redisClient.brpop: Redis connection gone from end event. [20:49:23] 2015-03-10T20:41:40.904Z - error: Caught error in redisClient.brpop: Redis connection to tools-redis:6379 failed - connect ECONNREFUSED [20:50:57] * legoktm is waiting for nagf to load [20:52:35] labs-morebots: still there? [20:52:35] I am a logbot running on tools-exec-13. [20:52:36] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [20:52:36] To log a message, type !log . [20:53:08] YuviPanda: redis problems? [20:55:11] I think it's redis [20:56:08] that's the only reason wikibugs and grrrit-wm would both get knocked offline [20:56:25] yeah, I know YuviPanda was concerned about redis yesterday, I just don’t know what the problem and/or solution is :) [20:56:31] Coren: same question? [20:57:02] yesterday saw "tools: Stop specifying params for redis maxmemory" [20:57:27] could it now hit maxmem because related to that? [20:57:45] https://gerrit.wikimedia.org/r/#/c/195237/ [20:58:15] hm, maybe — probably some tool is running amok and eating all the redis [21:05:40] I think that's also likely [21:18:57] andrewbogott: Sorry, just back from the passport office. Reading scrollback. [21:21:19] aaa sooo nice [21:22:33] Yeah, few to no redis users set ttl on their keys, so as it fills things get ejected lru. At this time, way too fast - the lifetime is roughly 3 days. [21:23:55] Coren: didn’t YuviPanda increase the memory size yesterday? [21:24:18] He did, afaik, but that only delays ejection. [21:25:12] Coren: yeah, I’m just surprised it’s crapping out again today already [21:29:04] Coren: wait, a full redis doesn’t explain failed - connect does it? [21:29:12] Or does redis just freak out when it’s full rather than throwing things overboard? [22:02:28] No, it's really not supposed to. It's just supposed to eject old keys [22:06:27] can we figure out who's filling up redis with so much data? [22:07:59] I'm trying, but Yuvi configured the server with no key listing. [22:08:41] sudo list-all-keys ;P [22:08:55] I'll have to snoop at the network level for a while. [22:09:20] Any significant outlier will be obvious quickly. [22:27:51] (03CR) 10Southparkfan: [C: 031] create wikisets.php config file [labs/tools/WMT] - 10https://gerrit.wikimedia.org/r/195659 (https://phabricator.wikimedia.org/T91941) (owner: 10John F. Lewis) [22:27:54] (03CR) 10John F. Lewis: [C: 032 V: 032] create wikisets.php config file [labs/tools/WMT] - 10https://gerrit.wikimedia.org/r/195659 (https://phabricator.wikimedia.org/T91941) (owner: 10John F. Lewis) [22:38:21] Coren: are there any docs for ipblocks_ipindex ? [22:40:27] Betacommand: Hm, I might not have written it down explicitly anywhere. It's the same principle as the others, only in this case the column that are indexed with rows elided are ipb_range_start and _end [23:26:49] 503 error on beta.wmflabs http://en.wikipedia.beta.wmflabs.org/wiki/Main_Page [23:52:16] Coren, YuviPanda ^ [23:52:29] (sorry if I'm pinging the wrong people ;-) [23:55:09] works for me [23:57:48] Hmm, it works if I'm logged-out, but not if I'm logged-in. [23:58:38] filed with paste of error message, and what I was doing at the time, at https://phabricator.wikimedia.org/T92347 [23:59:28] logging in there seems dangerous since it's just http [23:59:58] beta shold only be throw away accounts with unique creds