[06:14:54] Nice logstash is working, boo puppet freshness checks are broken.... I'll look at them when I get home later. [06:18:05] New patchset: DamianZaremba; "Tweak config for passive check" [labs/nagios-builder] (master) - https://gerrit.wikimedia.org/r/26933 [06:19:01] New patchset: DamianZaremba; "Tweak config for passive check + whitespace fix" [labs/nagios-builder] (master) - https://gerrit.wikimedia.org/r/26933 [06:22:12] Change merged: DamianZaremba; [labs/nagios-builder] (master) - https://gerrit.wikimedia.org/r/26933 [06:27:54] http://i-000003a9.pmtpa.wmflabs/#eyJzZWFyY2giOiJEZXBlbmRlbmN5IFBhY2thZ2UgXFxbLitcXF0gaGFzIGZhaWx1cmVzOiB0cnVlIiwiZmllbGRzIjpbIiJdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjE1IG1pbnV0ZXMiLCJncmFwaG1vZGUiOiJjb3VudCIsInN0YW1wIjoxMzQ5NTA0ODQ5MjkwfQ== that explains the dpkg issues then [06:55:25] !log nagios Fixed permissions on rw dir so snmptt can submit trap results. [06:55:26] Logged the message, Master [09:49:01] petan: Since you're one of the admin-people for bots, is there a reason I can sudo on bots-3 but not bots-1? [10:17:58] petan: Do you know? [11:58:42] is there someone around with labs access, who would restart COIBot for me? [11:58:45] (from #mediawiki) [12:02:06] It's on bots-3 according to the page [13:44:09] is there someone around with labs access, who would restart COIBot for me? [13:58:22] could do [13:58:54] sDrewth: Remind me who the owner is [13:59:02] Beetstra [13:59:09] the instructions are in the directory [13:59:14] yeah [13:59:25] slight issues with nfs being crap so I can't login right now :( [13:59:27] it runs in #wikimedia-external-links and has stopped registering links :-( [13:59:35] When it lets me in I'll restart it [13:59:36] k, thx for trying [13:59:44] :-))) [13:59:56] * Damianz uses andrewbogott_afk to tickle nfs in its sweet spot [14:00:37] hmm, depending where Werdna is, if at home it is midnight [14:02:13] I suspect it got caught by oomkiller [14:02:27] We need to make these instances bigger/make more instances, pending my work in puppet though sadly :( [14:03:56] !log bots bots-3 oom, killed beetstra's processes to get a gb of ram back [14:03:58] Logged the message, Master [14:05:27] !log bots bots-3 restarted linkwatcher, coibot, xlinkbot & unblockbot under beetstra's user [14:05:28] Logged the message, Master [14:05:31] sDrewth: Should be back [14:05:38] * sDrewth goes to look [14:06:00] lego|away: Probably because bots-1 use to host bots-labs stuff [14:06:43] it is there, and I think that it was more the linkwatchers that were not paying attention [14:07:02] so I will need to wait while it does its business and finds the evil bastards [14:11:43] Well at least you know what the bots' suppose to do :P [14:48:19] Damianz: Hello I would like to getting started with some pywikipedia bot on Labs... I do think I have shell access now. Could you please grant me bastion and bots access now? Thant would be nice! [14:51:21] !log bots Added DrTrigon as a member [14:51:23] Logged the message, Master [14:51:28] DrTrigon: Feel free to use bots-4 [14:52:10] Damianz: Cool thanks taht was fast! ;) Would it make sence to add me to 'Tools' too? [14:52:52] I don't think 'tools' has anything in it and I don't think I'm a member either so can't. [14:53:54] Damianz: ok... but thanks so far! I will search a member of that (but seems not to be that imporant... ;)) THANKS! [14:54:18] np [14:54:30] * Damianz goes back to finding clothes for next week and shopping [14:55:46] 10/06/2012 - 14:55:46 - Created a home directory for drtrigon in project(s): bots [15:00:42] Change on 12mediawiki a page Developer access was modified, changed by Vasu.raman link https://www.mediawiki.org/w/index.php?diff=591211 edit summary: [15:00:47] 10/06/2012 - 15:00:47 - User drtrigon may have been modified in LDAP or locally, updating key in project(s): bots [15:10:30] Change on 12mediawiki a page Developer access was modified, changed by Sharihareswara (WMF) link https://www.mediawiki.org/w/index.php?diff=591216 edit summary: /* User:Vasu.raman */ [15:10:49] 10/06/2012 - 15:10:48 - User drtrigon may have been modified in LDAP or locally, updating key in project(s): bots [15:11:00] 10/06/2012 - 15:11:00 - Updating keys for drtrigon at /export/keys/drtrigon [15:11:52] Change on 12mediawiki a page Developer access was modified, changed by Sharihareswara (WMF) link https://www.mediawiki.org/w/index.php?diff=591217 edit summary: /* User:Luft-on */ [15:12:51] Change on 12mediawiki a page Developer access was modified, changed by Sharihareswara (WMF) link https://www.mediawiki.org/w/index.php?diff=591218 edit summary: /* User:Zeljko.filipin(WMF) */ [15:14:03] Change on 12mediawiki a page Developer access was modified, changed by Sharihareswara (WMF) link https://www.mediawiki.org/w/index.php?diff=591219 edit summary: /* Requests */ All done! Thanks, Jeremy [16:05:46] 10/06/2012 - 16:05:46 - User louisdang may have been modified in LDAP or locally, updating key in project(s): hadoop [16:05:57] 10/06/2012 - 16:05:57 - Updating keys for louisdang at /export/keys/louisdang [16:19:39] GRRRRRRRRRRRR [16:21:08] how annoying [16:21:15] instance won't let me login and has no logs [16:29:02] Damianz: Is that currently a general problem for all instances? I also get permission denied when trying to log into an instance. [16:29:18] no [16:29:28] though awesome [16:29:35] rebooting instances seem to hate me [16:29:59] It's a random issue where key auth times out which seems to be related to nfs being shite [16:31:13] paravoid: You around atm? [16:35:18] Oh, my home directly doesn't seem to exist anymore. I guess that might explain why it doesn't like my ssh keys. [16:36:29] yeah [16:36:32] writing an email atm [17:11:18] Is something down right now? I'm not able to get into bots-1,-2,-3. or -5 [17:11:24] channel 0: open failed: administratively prohibited: open failed [17:11:24] ssh_exchange_identification: Connection closed by remote host [17:12:46] or -4* [17:14:01] short answer, yes [17:14:12] try the i- name [17:14:16] could be one of... 3 times [17:14:19] s/times/things/ [17:14:56] awesome [17:15:01] :( [17:15:07] i wish i could help [17:15:31] ok trying [17:15:47] LeslieCarr: You could supply cookies to Ryan after he kills nfs ;) [17:15:55] hehe [17:16:03] i'm more of a threatener type of person [17:16:12] Damianz: just ssh legoktm@i-000000a9.pmtpa.wmflabs ? [17:16:13] like "fix this or i knife you" [17:16:30] I tried that [17:16:32] That didn't take my public key [17:16:32] legoktm: yes [17:16:35] ok [17:16:39] so it's not dns being crap [17:16:42] probably nfs being crap [17:16:45] see mailing list [17:16:49] I saw [17:17:15] On the bright side, I think the instances have only 1 ip for the resolver which is the working one... I'd check but I can't actually ssh back in now :( [17:17:17] logged into bastion and got this: [17:17:19] Could not chdir to home directory /home/legoktm: No such file or directory [17:17:23] yeah that's nfs being crap [17:17:46] Oh well just the main anti vandal bot for enwiki down, dev instances un-accessible. I'll go back to washing clothes for my trip next week. [17:19:10] Erp. Well I'm not in a super hurry, I would just like to know why my bot died in the middle of the night [17:19:27] hi, can anyone look into this: my home directory is not getting mounted on bastion host. [17:20:39] ahh [17:21:07] legoktm: Might be the instance ooming a little which we'll sort in bots before the end of the month or stuff being a bit funky. [17:21:23] ooming? [17:21:28] running out of ram [17:21:44] * Damianz will finish puppetizing bots stuff when he gets back in so we can roll out new instances with an updated os, more ram and make things a little better [17:22:16] Ah gotcha [17:23:16] 'Why don't we just port MediaWiki to Java?' lolololololololololol [17:23:29] I swear the dev mailing list provides half my weekly entertainment [17:35:05] i'm going to text ryan and tell him shit's all borked [17:35:44] ryan is texted [17:35:47] I wasn't going to be as rude as to interrupt his saturday apart from emailing him :P [17:36:11] well i'm a rude woman [17:36:19] Typically the one weekend that all 3 people who could fix it are away stuff decides to bork. [17:36:20] as if you didn't know ;) [17:36:54] LeslieCarr: My image of you in the dc is one involving a cat 5 flogger and a dc tech bent over plugging a cable in, possible with a dog collar. [17:37:32] Damianz: while I am often informal, this is still my professional career, and I would like to keep the discussion back to a more professional level please [17:38:06] :P [17:39:19] * Damianz notes his actually image isn't that amusing but everytime you mention rude or mean that could easily be depicted for amusement. Apologies for any offence taken [17:40:05] thanks :) [17:47:03] It would be kind of nice if OpenSSH-LPK was do-able without patching openssh.. that combined with pam_mkdir and gluster would make things much more distributed and redundant. Though just another automount in gluster for keys would be shiny also. [17:47:46] no response from ryan - he is probably out partying it up [17:51:00] Partying it up at 10am sounds like a bad idea heh.. though I'll probably be doing that on Monday as it's just after I'm due to land in Dublin. [18:01:34] Damianz: i once had to deal with an outage (actually was a power outage for most of the colo and our single rack was completely down) at around this hour on a sunday (but 2pm not 11am: I'm Eastern) because the only person that did know how our stuff worked was already kinda drunk. kinda had an excuse though: it was the NYC marathon and supposedly traditionally people go drink while cheering people on? or something like that. don't really remembe [18:02:04] lol [18:02:34] the part I do remember was him giving me a list of what to bring up in what order and what cron daemon's to disable and how to query the facility about whether the power will be stable or not when it's back on [18:02:51] I once dealt with a dns outage drunk, called in at the office on the way home from the pub to grab my bag to find the new guy freaking out at dozens of hosts with random services down (seems ftp goes away if you kill its resolving dns servers). [18:04:11] * Damianz notes to go watch a marathon if it involves beer [18:07:41] hm [18:09:33] <^demon> It's a Ryan_Lane! [18:09:42] something's up with dns... [18:10:53] Ryan_Lane: Also my home directory didn't mount when I logged into bastion [18:10:55] Hi Ryan [18:11:00] yeah [18:11:02] this is due to dns [18:11:06] I assume you saw my email :) [18:11:09] Ah ok [18:12:34] nah. leslie paged me [18:12:45] LeslieCarr: thanks for paging me btw :) [18:12:47] sorry Ryan_Lane :-/ [18:13:00] homedirs are back [18:13:02] Ah [18:13:05] I wonder why dns failed [18:13:31] Clearly we need LeslieCarr to threaten knifing people more often :D [18:13:37] Thanks Ryan_Lane! [18:13:56] it only failed on labs-ns0 [18:13:59] not labs-ns1 [18:14:08] but it was still returning on the primary [18:14:34] legoktm: yw [18:14:41] I need to track down why this happened :( [18:16:35] it couldn't talk to ldap? [18:16:38] weird [18:17:30] Does it just talk to the general ldap servers? (generally I slave the db to each ns server for redundancy reasons) [18:17:50] ah [18:17:50] crap [18:17:57] it's only configured to talk to one server [18:18:02] =D [18:18:06] -_- [18:18:08] :-) [18:18:16] I thought I fixed that ages ago [18:18:26] I wouldn't be surprised if it's the only thing still configured that way ^.^ [18:18:34] well, I'm going to open a bug for that [18:18:43] well, there isn't too much talking to ldap [18:19:23] There's probably some script stuff either hardcoded or with outdated ldap configs around somewhere. [18:19:32] likely [18:19:37] well.... [18:19:37] no [18:19:44] they read from /etc [18:20:29] mhm [18:20:43] while you're fixing dns, could you change the SOA to a valid NS [18:21:21] which SOA isn't? [18:21:53] Both [18:21:57] They are set to virt0 still [18:22:04] Also when nothing else is broken could you un-break bots-cb (failed to re-appear after a reboot because nova hates me). [18:22:08] (Sorry for a todo list :)) [18:22:24] o.O [18:22:30] I could have sworn I fixed that [18:22:53] ah [18:23:07] this must have happened when I wiped out the config file for the wiki [18:23:19] :) [18:24:02] Our authoritive servers also do recursive lookups internal stuff, right? [18:24:18] only kind of [18:24:29] (If so all the resolve files only have 1 ns currently also, which makes sadtimes though luckly that didn't bork) [18:24:34] dnsmasq on the network node answers those directly [18:24:35] s/bork/borke/ [18:24:40] ah [18:24:45] then they hit the primaries [18:24:55] the primaries then fork that over to labs dns [18:25:14] the network node runs dns [18:25:14] wow... that's going around the houses a little [18:25:25] Hello sir, at what level would you like to impliment split horizons. [18:25:27] dnsmasq, anyway [18:25:49] Though as long as the network nodes are configured for both that's good... if the network node is down we have bigger issues :) [18:25:53] dnsmasq just forks off to the system dns [18:26:01] if it doesn't have an entry [18:26:16] the system dns for the virt hosts needs to be pointing to our recursor [18:26:24] labs dns doesn't have a recursor [18:26:48] the recursor looks up directly from labs dns [18:26:52] so it's not a ton of hops [18:27:14] It's mostly 'internal' then which is better than bouncing around recursive queries all over the place. [18:27:24] Kinda makes sense since 'labs' stuff there lives in 'prod' [18:27:34] yeah [18:27:59] ok. SOA record will fix itself on next dns change [18:37:42] :) [18:39:04] these deadlocks are killing me [18:39:17] in nova-compute [18:39:36] :( deadlocking is bad [18:39:39] yes [18:39:46] so, I'm rebooting your instance [18:39:50] it had the same problem [18:39:54] it wasn't defined in libvirt [18:39:58] Awesome [18:40:13] I'm going to run a script soon on all hosts to define everythng [18:40:20] How is that even possible considering it was just running on the host... bleh [18:40:28] it'll error out for almost everything, but that's fine [18:40:38] likely it was an instance we cold-migrated [18:40:47] and for some reason it didn't get defined properly [18:46:33] Damianz: Sorry for bothering you again; What do I have to do to get "/data/project/bot_name"? Where do I have to enter my 'bot_name' ? Thanks! [18:54:12] Hmm seems my instance didn't boot, maybe Ryan'll check his email later and sort it. No huge rush for now... [18:55:11] Damianz: ...did you talk tome...? [18:55:13] DrTrigon: sudo mkdir -p /data/project/ && sudo chown /data/project/ && cd /data/project/ && [18:55:14] Damianz, so is ClueBot NG (supposed) to be running on labs now? [18:55:38] Krenair: CBNG has run on labs for months and yes, but the instance is down right now [18:56:00] Damianz: not auto-mount?? [18:56:32] DrTrigon: /data/project is an automount gluster brick, everything under it is 'project storage' and as such shared by project members. [18:57:02] It's treated just like a normal dir and is magically global to all project instances. [18:57:33] ok - got it! ;) thanks! other question; what to do to get e.g. python 2.7? it's not in the repos... or is it? [18:57:43] * Krenair keeps an eye on Huggle [19:00:00] DrTrigon: currently unless you're on a updated instance it's not avaible. Possible a package for it. [19:00:29] Krenair: Just don't keep an eye on Stiki as CBNG's not-reverted-high-scoring-vandalism is one of the main inputs heh [19:00:43] when does it get updated? may we do it ourselves? ;) [19:00:51] short answer, no [19:01:16] We'll probably migrate them all in the next month and introduce more 'restricted' instances. [19:01:30] Currently a little messy in setup and lacking in ram in parts. [19:02:16] If you want to use bots-nr1 it's running precise that's fine... but you can't install anything on it currently. [19:02:32] * Damianz really needs to finish the puppet stuff [19:03:08] Ryan_Lane: Hi again, that instance still hasn't re-appeared :( Probably failed to boot for some reason but the console log shows nuffin [19:03:52] Damianz: ok thanks so far - I am on the svn checkout of my code... good night! ;) [19:04:08] No problem :) [19:05:36] Is it possible to ssh / scp from one instance to another? I can access both instances from bastion, but get a "Permission denied (publickey)" if I try and ssh directly between two instances. [19:06:17] apmon: Not easily, you can using bastion as a proxy. [19:06:24] If they are in the same project I'd suggest project storage. [19:07:04] trying now [19:07:11] it may have timed out [19:07:57] Oh, yes I forgot that project storage is shared. That makes transferring files easier than scping them over via bastion... [19:09:09] yeah [19:09:18] project storage is way more convenient than scp ;) [19:10:13] * apmon is copying over some puppet scripts to a new instance to verify that they actually work on a fresh instance [19:11:12] If you like puppet feel free to path puppetmaster::self to support using project storage with a custom dir :D [19:13:35] wow. wtf nova. pick up the damn job [19:15:39] hm. something else is causing it to be undefined. [19:18:46] I think nova has a secret setting called $make_life_a_pain [19:19:51] yes [19:23:46] Damianz: instance is up now [19:25:27] Yay, thanks. [19:30:09] !log bots checkout of DrTrigonBot framework to /data/project/DrTrigonBot; svn co https://svn.toolserver.org/svnroot/drtrigon/ . [19:31:16] Hey that is space seperated, theres a space at the end... though yours is nearer and could probably be done with <%= ldap_hosts.join(" ") %> thinking about it. [19:31:23] s/nearer/neater/ [19:36:30] * Damianz notes he is mostly an ass [19:38:45] Damianz: What do I have to do to get the '!log' working here? :) [19:39:01] !log bots checkout of DrTrigonBot framework to /data/project/DrTrigonBot; svn co svn.toolserver.org/svnroot/drtrigon/ . [19:39:03] Logged the message, Master [19:39:05] He has a thing about urls [19:39:14] dunno why, it's stupid [19:39:46] * Damianz might convince ^demon to move it to git and submit a patch for review [19:40:11] but he did not accept other tries of me without the url included... he ignores me... ;) [19:40:53] (thanks for logging btw. ;) [19:41:10] Pretty sure it doesn't [19:41:17] It does randomly fail to edit the page though [19:41:42] Something that needs some time in better logging/error handling/general not dissapearing but like everything else time gets used quickly. [19:41:53] do I hav to do it here? I did in a private chat with labs-morebots... [19:42:12] yeah [19:42:18] it won't respond to pm AFAIK [19:42:55] ...aaa I see! thanks again!!! ;) [19:43:29] You know if we wrote a plugin for SAL (chat a while back) we could actually make it avaible as an rss feed and do awesome [19:44:00] ;))) would be cool - indeed...! :) [19:44:37] Also would make the horrid current SAL page better (if you've ever looked at that your eyes will bleed) [19:57:14] !log centralauth Add centralauth-daemons [19:57:14] Logged the message, Master [19:57:14] !log centralauth Move memcached from centralauth-frontend to centralauth-daemons [19:57:14] Logged the message, Master [19:57:15] RogueMadman: Ryan might be, probably best to email the list as it's a saturday afternoon. [19:57:22] I need to make a request queue [19:57:24] email the list [19:57:33] I'd do it right now, but I'm heading out the door [19:57:46] Yeah, like shell requests would be awesome [19:58:04] yep [19:58:10] Good point. Thanks. :D [19:58:21] Clearly you need Jira. [19:58:25] Sorry, it just had to be said. xD [19:58:34] we have bugzilla [19:58:35] Feel free to use BZ if it makes you feel better [19:59:09] Brb need to throw some clothes in the wash [19:59:25] I know; I was being facetious. [19:59:36] :p [19:59:36] :D [20:00:19] I'm sure we could get someone to write an extension to use a mediawiki based bug tracking system to replace bz with *shudder* [20:03:01] aude: did you already create a git repository for the maps project? [20:04:51] apmon: i requested one [20:05:31] Do you know what the next step is to get the wikimedia-mobile osm server setup? [20:05:33] apmon: not created yet [20:05:45] we can get stuff into git [20:06:19] maybe if you're up to it, we could schedule a time to "meet" with the mobile people? [20:06:26] e.g. irc or whatever [20:07:06] https://www.mediawiki.org/wiki/Git/New_repositories/Requests [20:07:33] I should be able to do that [20:07:44] ok [20:08:10] * aude saw your email, so that should be good to poke them [20:09:17] The current puppet scripts are in /data/project/tmp and the debian packages in /data/project/repo if you want to take a look at them [20:09:21] ok [20:09:29] thanks [20:17:33] Any chance someone who has access to sudo on bots-1 would be willing to install git-core? [20:20:19] Change on 12mediawiki a page Developer access was modified, changed by West.andrew.g link https://www.mediawiki.org/w/index.php?diff=591252 edit summary: [20:22:13] :D [20:22:24] Now I have to request that project so I'm not slacking. [20:22:25] :p [20:29:45] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=591255 edit summary: /* User:West.andrew.g */ done [20:30:31] RogueMadman: does he IRC? [20:30:50] jeremyb: I don't think so but I can e-mail him easily. [20:31:02] He made that request an hour after I asked him to. :x [20:31:03] RogueMadman: he needs to request shell obviously [20:31:19] Ah. And needs to do a public key and the like? [20:31:27] sure [20:31:54] He uploaded a public key for his Toolserver request which has been ignored. :x But I'll contact him and ask if he wants to IRC or something else. [20:32:20] If he doesn't get on IRC, what information should I request from him at a minimum? [20:41:46] @seen Hydriz [20:41:46] Jan_Luca: Last time I saw Hydriz they were quiting the network N/A at 10/6/2012 4:50:47 AM (15:50:58.8209040 ago) [20:42:26] Because we really need the fractions of seconds. ;x [20:45:28] Do someone know a good puppet tutorial? [20:53:09] So I stuck https://labsconsole.wikimedia.org/wiki/Help:Contents#Requesting_A_New_Project together, seems to work [20:53:36] * RogueMadman logs in. [20:53:43] Jan_Luca: http://docs.puppetlabs.com/learning/? [20:54:22] Oh. My old phone with Google Authenticator got wiped. [20:54:24] Hmmm. [20:54:31] Hope I can find those recovery tokens. ;x [20:54:31] Damianz: [03:17:33 PM] Any chance someone who has access to sudo on bots-1 would be willing to install git-core? [20:54:41] just did it [20:54:44] thanks [20:54:47] and was about to reply [20:54:49] DAMN YOU [20:54:52] :) [20:55:01] !log bots Damianz installed git-core on bots-1 [20:55:03] Logged the message, Master [20:55:21] Hey, I did find the recovery keys. [20:55:23] I'm smart. :D [20:55:24] RogueMadman: If you sell your soul we could probably wipe those if needed. [20:55:33] Damianz: Thank you [20:56:51] I submitted a new project request. :D [21:09:16] I totally didn't associated madman to you, though considering I said to post to the mailing list earlier I really should have heh. [21:10:57] Haha. [21:11:07] Yeah, the nick Madman's taken on Freenode, unsurprisingly. D: [21:34:06] kl [21:36:35] New patchset: jan; "Add .gitreview and .gitignore" [labs/centralauth] (shared-wiki-config) - https://gerrit.wikimedia.org/r/27013 [21:37:13] Change merged: jan; [labs/centralauth] (shared-wiki-config) - https://gerrit.wikimedia.org/r/27013 [21:39:06] New patchset: jan; "Import shared wiki config" [labs/centralauth] (shared-wiki-config) - https://gerrit.wikimedia.org/r/27014 [21:39:40] Change merged: jan; [labs/centralauth] (shared-wiki-config) - https://gerrit.wikimedia.org/r/27014 [21:42:11] New patchset: jan; "Add .gitreview and .gitignore" [labs/centralauth] (master) - https://gerrit.wikimedia.org/r/27016 [21:42:42] Change merged: jan; [labs/centralauth] (master) - https://gerrit.wikimedia.org/r/27016 [21:44:41] https://pbs.twimg.com/media/A4N8ZVPCQAAKiVr.jpg lol [21:45:35] HAHAHA. [21:45:41] lame [22:09:18] Change on 12mediawiki a page Developer access was modified, changed by Carlb link https://www.mediawiki.org/w/index.php?diff=591269 edit summary: [22:26:28] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=591272 edit summary: /* User:Carlb */ not done [22:27:02] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=591273 edit summary: /* User:Carlb */ fmt [22:39:36] Change on 12mediawiki a page Developer access was modified, changed by Carlb link https://www.mediawiki.org/w/index.php?diff=591275 edit summary: /* User:Carlb */ - set shell name to SUL identity [22:56:09] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=591277 edit summary: /* User:Carlb */ done [23:31:32] jeremyb, when are we making labs open again? I remember Ryan saying something about a change that needed merging that should be done soon?