[00:00:43] Coren, I have a non-urgent question if you are still working [00:00:59] I'm semi-working as I usually do evenings. What be up? [00:01:24] In short... [00:01:25] # mount /data/project [00:01:26] mount.nfs: mounting labstore.svc.eqiad.wmnet:/project/extdist/project failed, reason given by server: No such file or directory [00:01:33] on instance extdist2. [00:01:37] I can see the volume on labstore1001. [00:01:43] And I can see the export [00:02:05] Not sure what to look for next [00:02:45] Huh. Odd. [00:03:19] Yuvi says that another instance in that project has the same complaint. [00:03:26] I see three IPs in the export. One of those extdist2? [00:03:46] 10.68.17.211 [00:03:48] 10.68.127.183, 200 and 211 [00:03:52] But, for my education, where are you seeing those? [00:04:05] 'exportfs' on labstore [00:04:08] um… is the export really 127.211? [00:04:16] No, .17 [00:04:19] Tpyo. [00:04:24] ok, that's right then [00:04:50] [16:56:27] legoktm: may I reboot extdist2? <-- go for it [00:04:59] * legoktm logs out [00:05:17] legoktm: ok, I'm going to hold off on the reboot until Coren is out of ideas... [00:05:51] andrewbogott: I can see no reason why the mount could fail; but I've seen very rare cases where the kernel seems to cache the negative hit on the mount because it managed to try it before the export was in place? [00:06:07] Though I haven't seen that happen in ages. [00:06:09] Would that caching be on the client or the server? [00:06:15] Yeah, I thought that was resolved somehow... [00:07:18] Try to reboot the instance; if it was client-side it should work. [00:07:23] And I'll be annoyed. [00:07:26] ok [00:08:11] I suppose it's still possible for the puppet run to outrace the manage-nfs-volumes daemon. [00:10:48] :( Sure enough, fixed by reboot. [00:11:22] Bah. I don't see any way of preventing this possible race other that putting some nasty interlock in puppet to force it to wait until the volume is ready. [00:11:27] And I would hate that. [00:14:11] It's ok -- just, next time someone complains about this I'll tell 'em to reboot :) [00:14:20] It must be quite rare. [00:17:23] Mind you, the most common cause of that error is "project storage not turned on" :-) [00:28:36] 3Wikimedia Labs: Cannot log into nor reboot social-tools1 instance "The requested host does not exist. " - 10https://bugzilla.wikimedia.org/67545#c2 (10Kunal Mehta (Legoktm)) Thanks, I totally forgot that it never got migrated. Everything works fine now :) [04:55:06] 3Wikimedia Labs / 3Infrastructure: Database upgrade MariaDB 10: Lock wait timeouts / deadlocks in a row - 10https://bugzilla.wikimedia.org/68753#c1 (10Sean Pringle) (In reply to metatron from comment #0) > - Is this behaviour transient? If not, please make it go away. Hopefully transient, yes. We are using... [05:44:33] drunken labs :) [07:13:57] !channels [07:13:57] | #wikimedia-labs-nagios #wikimedia-labs-offtopic #wikimedia-labs-requests [07:18:03] what?, no #wikimedia-labs-mad-scientists ? [08:03:56] @notify hashar [08:03:56] I'll let you know when I see hashar around here [08:57:15] does anyone know the status of the new storage for dumps etc ? [09:44:33] beta cluster dead? [10:05:54] TBloemink: yes [10:06:00] hashar_: fix it [10:06:01] darn [10:08:19] hashar_: why you removed me from project admin role :( [10:08:28] I created that project D: [10:21:24] has it been reported yet that http://commons.wikimedia.beta.wmflabs.org/ seems to be down? [10:24:39] hashar poke [10:24:52] yes dan-nl the beta cluster is down [10:25:01] thanks [10:36:17] bah [10:36:34] that is hhvm [10:37:05] !log deployment-prep Restarted hhvm on mediawiki{01,02} [10:37:07] Logged the message, Master [10:37:17] dan-nl: TBloemink petan beta cluster should be back now [10:37:33] s/should/might/ [10:37:35] hashar: why you removed me from project admins D: [10:37:36] nice [10:37:42] how dare you! [10:37:43] :D [10:37:47] petan: so you don't get root on beta cluster :D [10:37:53] it was my project once [10:37:54] we can grant sudo on some boxes if needed though [10:37:57] I created it :/ [10:38:04] it's up and running again, thanks! [10:38:24] petan: yeah and thanks for that. But the idea has been essentially took over by WMF and matches production more closely [10:38:31] dan-nl: thanks for confirming! [10:38:39] bah... damn WMF [10:38:43] dan-nl: the reason is that we run hhvm and ... hhvm is broken [10:39:01] petan: see the foundation has an utility to serve the community :-] [10:39:26] I think that foundation is just good in overtaking things... just like microsoft :P [10:39:49] the difference being that as a shareholder of Microsoft you can barely be heard [10:40:01] hold on [10:40:03] whereas at WMF as a community member your voice is heard [10:40:10] foundation has shareholders? o.O [10:40:14] :D [10:40:19] of course [10:40:28] not in the sense of people holding stocks [10:40:36] see it as: [10:40:50] but holding stocks can be more fun :P [10:41:02] community --- (elect) --> board --- (designate) ---> executive director --- (hire) ---> staff & contractors [10:41:26] if we had stock I would have a bunch of millions of dollars and would be bored on some SF beach [10:41:42] much funnier to do stuff instead :-] [10:41:57] hmm, HHVM is an open-source virtual machine designed for executing programs written in Hack and PHP … Hack = Programming productivity without breaking things ... [10:42:03] and the whole project would have been entirely different anyway since stock holders expect dividends to be yield and you become a for profit org [10:42:05] I see it more like: community --- (invent something) ---> foundation ---> (overtake that something) ---> community ----> (send money to foundation for something they once invented) [10:42:22] dan-nl: yeah HHVM is basically a reimplementation of the PHP language. It has been written by Facebook [10:42:39] dan-nl: that yields huge perf improvements (something like twice the perfs) [10:42:59] dan-nl: hack is a superset of the PHP language code. The most obvious one being to make PHP a type strict language [10:43:13] so true == 1 would yield a type error (iirc) [10:43:30] instead of PHP that evaluates it as True [10:43:44] but true == 1 should evaluate as true D: [10:43:44] ah, i see … so someone didn't like php's dynamic typing [10:43:49] XD [10:43:57] are they planning to re-write js as well? [10:44:03] :) [10:44:11] petan: nice, that's like the circle of big pharma when they patent indigenous medicaments [10:44:15] well glad to see there's a performance boost [10:44:39] petan: na community invents stuff. And at some point foundation throws staff at it :-] [10:44:42] Nemo_bis: yep :P [10:45:01] hashar: and removes access from community people :P [10:45:11] hmm, Nemo_bis that tagent yields the idea that one day you'll have to pay to be born because you won't your own genes anymore [10:45:15] so that only staff can fix it when it breaks [10:45:31] dan-nl: yes, they're actually doing this [10:45:36] poor mother nature, she really needs to find a lawyer fast [10:45:43] dan-nl: a corporation sued someone for using their own genes [10:45:50] lol [10:46:05] future is bright [10:46:17] i can imagine, it's a sign of a democratic repuclic gone awry [10:46:18] IIRC *that* was overturn by the Supreme Court, but I wouldn't be so sure [10:46:28] well you already pay taxes for breathing fresh air on some places [10:46:49] dan-nl: you can learn a bit more about hhvm at https://en.wikipedia.org/wiki/HipHop_for_PHP [10:46:59] dan-nl: there is a section explaining the virtual machine [10:47:05] cool, thanks! [10:47:20] HHVM converts PHP code into a high-level bytecode [..] This bytecode is then translated into x64 machine code dynamically at runtime by a just-in-time (JIT) compiler. In these respects, HHVM has similarities to virtual machines for other languages including C#/CLR and Java/JVM. [10:48:15] nice, at a quick glance it sounds like a good move [10:48:43] petan and dan-nl, Vandana Shiva is a very nice read on these topics (doesn't give a complete representation of all views but still useful and enjoyable) [10:50:11] nice, thanks Nemo_bis [10:50:12] dan-nl: and to understand what the VM is doing : https://www.facebook.com/notes/facebook-engineering/the-hiphop-virtual-machine/10150415177928920 [10:50:21] dan-nl: that is a public post, does not need to be logged with facebook [10:52:24] so hhvm is also the virtual machine the foundation will use for its wikis? [10:52:52] that would mean a re-write or at least an examination of current code to make any necessary changes - correct? [10:53:26] really interesting read ... [10:57:27] dan-nl: yeah that is what we are doing [10:57:38] a bunch of folks have been working on it :] [10:57:44] such as migrating our C extensions [10:57:59] cool [10:58:08] (actually Tim Starling rewrote some part of HHVM to supports extensions code ) [10:58:15] + we run tests of extensions [11:20:50] hashar: any idea why, http://language-stage.wmflabs.org/ is down? hhvm there too (on labs)? [11:27:45] anyone know why interwiki links are not present on commons beta? [11:45:59] kart_: no idea I don't have access to that so you are on your own :-] [11:46:23] dan-nl: i think we have a bug about setting interwikis properly on beta cluster [11:46:29] I think they use whatever is in production [11:58:08] hashar: reboot worked :) [11:58:34] does https://graphite.wikimedia.org work for anybody here? i cannot log in with the user+pass I use at https://wikitech.wikimedia.org/w/index.php?title=Special:UserLogin [12:01:54] hashar, okay i'll see if i can find the bug. the one specific interwiki link i'm looking for on beta is commons, but maybe it's not in the table because it would refer to itself on production … [12:02:27] dan-nl: https://bugzilla.wikimedia.org/show_bug.cgi?id=67931 Beta should not use productions interwiki.cdb [12:03:02] dnaber: access to graphite is restricted to people having an account in the 'wmf' ldap group [12:03:09] dnaber: i.e. staff / contractors [12:03:22] dnaber: though gdash expose a bunch of the data publicly [12:07:19] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/Roadmap en was modified, changed by Dnaber link https://www.mediawiki.org/w/index.php?diff=1082259 edit summary: make clearer who can access graphite [12:10:05] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/Needed Toolserver features was modified, changed by Dnaber link https://www.mediawiki.org/w/index.php?diff=1082261 edit summary: /* Logs/Stats */ make clearer who can access graphite [12:25:30] dnaber: if you're looking for graphite for toollabs, it's at graphite.wmflabs.org. But it's only experimental at this point, so don't depend on it fully yet [12:29:09] YuviPanda: I'm looking for simple stats (http requests or so) for http://tools.wmflabs.org/languagetool, graphite doesn't seem to have anything about my tool . I guess I need to activate it? but i'm not using lighthttpd, but tomcat. [12:29:35] dnaber: ah, no, we don't have http requests yet, at all, sorry [12:29:38] dnaber: hence 'experimental' [12:29:58] dnaber: you should have tomcat log access requests and parse them yourself at this point. hopefully we'll have better stats in a month or so [13:02:52] 3Wikimedia Labs / 3tools: Rename revision_userindex to revision - 10https://bugzilla.wikimedia.org/66786#c4 (10Marc A. Pelletier) The naming convention was selected on the "least surprise" principle; there are a number of rows not available in the _%index views and those rows would be missing without explana... [13:04:36] 3Wikimedia Labs / 3tools: Rename revision_userindex to revision - 10https://bugzilla.wikimedia.org/66786#c5 (10Yuvi Panda) Yeah, I think the rename will actually give people the 'least surprise' - I think running a query you expect to be fast but ends up being super slow and then having to go dig around why... [13:21:51] 3Wikimedia Labs / 3tools: Rename revision_userindex to revision - 10https://bugzilla.wikimedia.org/66786#c6 (10Marc A. Pelletier) That's not what I meant (performance); currently selection on 'revision' will give you the same result rows as making that same query in production whereas use of 'revision_userin... [13:30:34] @jb ToAruShiroiNeko [13:42:13] It looks like someone is running SmartAdmin on Tool Labs [13:45:43] rillke: what's smartadmin? [13:45:50] https://wrapbootstrap.com/theme/smartadmin-responsive-webapp-WB0573SK0 [13:46:26] oh [13:46:32] rillke: poke Coren and tell him? [13:46:38] it was looking so professional and good [13:46:59] that it couldn't be open source .... [13:47:01] tsk, tsk. Where? [13:47:09] http://tools.wmflabs.org/oqtools/ [13:47:24] It /is/ very nice looking. [13:47:51] heh [13:47:56] It's also clearly an install with default test/sample data. :-) [13:50:36] 3Wikimedia Labs / 3tools: Rename revision_userindex to revision - 10https://bugzilla.wikimedia.org/66786#c7 (10Aaron Halfaker) Let me just say that I'm very surprised that either of the tables is missing rows that the other has. Why would "userindex" suggest that some rows have been filtered? [13:53:51] 3Wikimedia Labs / 3tools: Rename revision_userindex to revision - 10https://bugzilla.wikimedia.org/66786#c8 (10Marc A. Pelletier) The rows that are filtered in the _userindex views are where the user id and user named have been NULLed because of suppression. You'd never *notice* the missing rows when using... [13:59:13] rillke: Proprietary software disabled and user notified/warned. [13:59:34] Thanks for noticing and reporting. [14:02:08] 3Wikimedia Labs / 3tools: Rename revision_userindex to revision - 10https://bugzilla.wikimedia.org/66786#c9 (10Aaron Halfaker) Marc, as a regular user of this data, that's very unintuitive. Also: SELECT COUNT(*) FROM revision_userindex WHERE rev_user IS NULL; [14:03:07] 3Wikimedia Labs / 3tools: Rename revision_userindex to revision - 10https://bugzilla.wikimedia.org/66786#c10 (10Marc A. Pelletier) That last paragraph should have ended "... if you are using the revision table, but mysteriously missing if you are using revision_userindex". [14:03:24] 3Wikimedia Labs / 3tools: Rename revision_userindex to revision - 10https://bugzilla.wikimedia.org/66786#c11 (10Marc A. Pelletier) rev_user may be NULL for reasons /other/ that suppression. :-) [14:07:38] 3Wikimedia Labs / 3tools: Rename revision_userindex to revision - 10https://bugzilla.wikimedia.org/66786#c12 (10Marc A. Pelletier) (In reply to Aaron Halfaker from comment #9) > Marc, as a regular user of this data, that's very unintuitive. Indeed it is, albeit clearly documented at: https://wikitech.wikim... [14:07:55] Coren: we should still rename it, I think [14:08:20] Aaron's confusion illustrates why I felt it was a bad idea to. :-) [14:09:15] People want correct results more than they want fast queries. :-) [14:10:29] the number of people hit by the performance isssue would be grater than the number of people hit by missign entries due to suppression, I think? [14:11:21] 3Wikimedia Labs / 3tools: Rename revision_userindex to revision - 10https://bugzilla.wikimedia.org/66786#c13 (10Betacommand) Honestly this is a case where we just need to train our users to use the correct view. Nothing should be changed on the server side. As it should reflect production on production table... [14:12:12] Coren: I think we just need a clearer documentation/support system [14:12:24] /Possibly/, which is why I am not entirely opposed to the idea -- but note how Aaron didn't even know there was a distinction in the data. In my case, failiure to read the doc can lead to slow queries and an opportunity to improve efficiency. In the other case, data is silently hidden. [14:13:01] Betacommand: That may well be. Our documentation is pretty complete, but is damn near veering into tl;dr territory. [14:13:52] Coren: depends on your perspective. I know finding labs/tools specific info on wikitech is a bitch [14:14:10] and that is spread across three wikis [14:14:25] meta, wikitech, and mediawiki [14:14:35] Betacommand: What we need is a technical writer to collect and organize what we have. [14:15:11] Not denying that [14:15:38] Because yeah, everything is pretty well documenting *somwhere* but it's either buried deep in walls of text or squirelled away in a dark corner that made sense at the time. [14:15:46] I know finding the existing docs for toolserver where fairly easy as it was located on the one wiki [14:16:42] and they where not mixed with documentation for other stuff [14:18:06] Let's just say that I agree with you that wikitech serving as "documents prod and internal WMF stuff" and "documents Labs" as the same time may not have been one of the most inspired ideas. :-) [14:18:21] it would be helpful to have something like https://www.mediawiki.org/wiki/Manual:Database_layout with listed key for views and all included extensions like wikidata [14:18:24] Coren: yeah, and Toollabs help page is definitely TLDR at this point [14:18:31] (a.k.a. polite way of saying "fooking stoopid") [14:18:58] hashar: beta is down again, fix it [14:19:35] (I would fix it myself but /someone/ revoked my permissions to do that) [14:20:20] @labs-project-instances deployment-prep [14:20:20] Following instances are in this project: deployment-apache02, deployment-cache-bits01, deployment-cache-mobile03, deployment-memc02, deployment-memc03, deployment-cache-text02, deployment-parsoid04, deployment-eventlogging02, deployment-bastion, deployment-apache01, deployment-salt, deployment-logstash1, deployment-parsoidcache01, deployment-redis01, deployment-upload, deployment-fluoride, deployment-db1, deployment-jobrunner01, deployment-videoscaler01, deployment-elastic01, deployment-elastic02, deployment-elastic03, deployment-elastic04, deployment-analytics01, deployment-cache-upload02, deployment-graphite, deployment-rsync01, deployment-memc04, deployment-memc05, deployment-db2, deployment-stream, deployment-pdf01, deployment-cxserver01, deployment-mediawiki01, deployment-mediawiki02, [14:20:57] Coren: what are your thoughts of moving it to http://tools.wmflabs.org/wiki/ [14:22:01] Betacommand: If we went that way (I'd have to float the idea internally first) I would absolutely want it to be a normal prod wiki and not yet another unsynced local install. [14:22:23] Betacommand: A priori though, I very much agree that having a documentation wiki for Labs (and just Labs) would be a big win. [14:22:37] <^d> yay more wikis. [14:23:08] Coren: I would agree about the prod format. and thats why I picked the uniform URL :P [14:23:42] Yeah, but I don't mean just prod-like; I meant part of the regular cluster. :-) [14:24:07] So can't be 'tools.wmflabs.org' :-) [14:24:48] Coren: why not? [14:25:15] create a specific re-write or what not [14:25:40] it makes the most logical sense to place it there [14:25:54] @notify hashar [14:25:54] This user is now online in #wikimedia-dev. I'll let you know when they show some activity (talk, etc.) [14:27:54] Betacommand: even if it's a rewrite, it'll have to go through the toollabs proxy, and hence won't be 'prod' [14:58:27] puppet.log: ESC[1;31mError: /Stage[main]/Role::Labs::Instance/Mount[/home]: Could not evaluate: Execution of '/bin/mount /home' returned 32: mount.nfs: mounting labstore.svc.eqiad.wmnet:/project/pagemigration/home failed, reason given by server: No such file or directory [15:00:21] !log deployment-prep Apache stuck with 65 children on both deployment-mediawiki servers [15:00:24] Logged the message, Master [15:02:40] Sob. $ grep "Notice:" /var/log/puppet.log | sort -u | wc 63002 441235 10669718 [15:13:49] beta seems to be spinning forever [15:17:12] !log deployment-prep _joe_ restarting hhvm on deployment-mediawiki01 [15:17:14] Logged the message, Master [15:27:17] [07:58:27] puppet.log: ESC[1;31mError: /Stage[main]/Role::Labs::Instance/Mount[/home]: Could not evaluate: Execution of '/bin/mount /home' returned 32: mount.nfs: mounting labstore.svc.eqiad.wmnet:/project/pagemigration/home failed, reason given by server: No such file or directory <-- did you try rebooting the instance? fixed it for my instance [15:30:52] 3Wikimedia Labs / 3Infrastructure: Database upgrade MariaDB 10: Lock wait timeouts / deadlocks in a row - 10https://bugzilla.wikimedia.org/68753#c2 (10Tim Landscheidt) (In reply to Sean Pringle from comment #1) > [...] > Today we are trialing READ-COMMITTED isolation level which was initially > REPEATABLE-R... [15:31:51] legoktm: yeah, I probably will now; puppet was stuck for hours changing permissions to files [15:31:51] 3Wikimedia Labs / 3Infrastructure: Database upgrade MariaDB 10: Lock wait timeouts / deadlocks in a row - 10https://bugzilla.wikimedia.org/68753#c3 (10Tim Landscheidt) *tools [15:32:12] When find is done in a couple seconds btw $ find /srv/vagrant/mediawiki/ ! -group 33 -exec sudo chown vagrant:www-data {} + [15:34:04] Nemo_bis: Ruby + sequential execution, IIRC :-). [15:50:08] !log deployment-prep installed libevent-dbg on deployment-mediawiki02 to capture an hhvm backtrace [15:50:12] Logged the message, Master [15:52:05] \o/ someone logging besides me [15:52:25] hehe [15:52:43] !log deployment-prep restarted hhvm on deployment-mediawiki02 [15:52:46] Logged the message, Master [15:53:33] 02 is responding now but 01 is still stuck [15:58:42] !log deployment-prep restarted hhvm on deploymnet-mediawiki01 [15:58:43] Logged the message, Master [16:00:08] 3Wikimedia Labs / 3Infrastructure: Database upgrade MariaDB 10: Lock wait timeouts / deadlocks in a row - 10https://bugzilla.wikimedia.org/68753#c4 (10metatron) (In reply to Sean Pringle from comment #1) Though I didn't change anything, the massive lock wait timeouts or deadlocks disappeared. Right now I'm... [17:08:54] I need to change the primary email associated with my jenkins account -- who do I poke to do that? [17:09:02] *not jenkins -- gerrit [17:10:51] mwalker: Isn't this coupled to wikitech's preferences? [17:12:28] maybe [17:12:31] I'll try [17:24:59] that was in fact the correct answer [17:30:51] 3Wikimedia Labs / 3deployment-prep (beta): Unable to upload new version of images in commons beta lab - 10https://bugzilla.wikimedia.org/68760#c1 (10Greg Grossmeier) p:5Unprio>3Normal Showing my ignorance: Do you have the right permissions? I apparently don't as I see "You cannot overwrite this file." on... [17:33:07] 3Wikimedia Labs / 3deployment-prep (beta): Unable to upload new version of images in commons beta lab - 10https://bugzilla.wikimedia.org/68760#c2 (10Chris McMahon) Let me know your user names and I can upgrade your privileges on beta labs. [17:44:51] 3Wikimedia Labs / 3deployment-prep (beta): Unable to upload new version of images in commons beta lab - 10https://bugzilla.wikimedia.org/68760#c3 (10Greg Grossmeier) (In reply to Chris McMahon from comment #2) > Let me know your user names and I can upgrade your privileges on beta labs. "Greg (WMF)" [17:47:39] !ping [17:47:40] !pong [18:35:22] 3Wikimedia Labs / 3deployment-prep (beta): Unable to upload new version of images in commons beta lab - 10https://bugzilla.wikimedia.org/68760#c4 (10Vikas) (In reply to Chris McMahon from comment #2) > Let me know your user names and I can upgrade your privileges on beta labs. LanguageScreenshotBot [18:38:51] 3Wikimedia Labs / 3deployment-prep (beta): Unable to upload new version of images in commons beta lab - 10https://bugzilla.wikimedia.org/68760#c5 (10Vikas) Looks like the images which were uploaded before(i.e. before we got the some problems with beta labs recently) have this problem. I tried updating the im... [18:56:49] GerardM-: I have some (annoying) news for the dump storage box: wipe still in progress but not done. Expected tomorrow. [19:09:10] !log deployment-prep deployment-bastion jenkins slave is stuck. Beta cluster is no more updating code :-// [19:09:13] Logged the message, Master [19:10:19] andrewbogott: Coren hmm, on https://wikitech.wikimedia.org/wiki/Special:NovaInstance I see nothing for graphite? [19:10:30] I'm pretty sure there are a couple of instances there [19:11:04] hashar: Often I've found if when that happens that just killing off all of the queued beta jobs will fix things [19:11:19] bd808: yeah something is terribly borked in J nekins :-// [19:11:29] There's some kind of deadlock that can happen and then they get stuck [19:11:41] :-( [19:12:14] YuviPanda: yeah, something is messed up; I'll look. [19:12:23] andrewbogott: ok! [19:12:37] YuviPanda: do you need it right now? i'd like to watch the end of the hhvm brownbag [19:12:57] andrewbogott: yeah, after that is fine. the new graphite box is almost done, so I want to apply the role on a trusty machine to test things with [19:13:11] bd808: I did just that :D [19:13:18] bd808: and restarted the slave connection [19:14:02] pff [19:14:22] might be solved. I am off again [19:14:25] YuviPanda: Logging out and in fixed it for me. [19:14:44] !log deployment-prep Removed all jobs from queue, restarted slave agent. Update Jobs coming back [19:14:47] Logged the message, Master [19:18:38] (03PS3) 10Jforrester: Add Beta Labs bugs for the QA channel [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/143239 [19:19:00] (03CR) 10Jforrester: "Lack of discussion is dispiriting…" [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/143238 (owner: 10Jforrester) [19:24:05] coren what is most annoying is not knowing ... Thanks for letting me know [19:25:26] andrewbogott: logging out and back in fixed it [19:25:37] hm, ok. [20:31:56] Could pr_index table on wikisource wikis be included in the tool labs sql replicas? [20:32:14] bawolff: poke Coren and file a bug? [20:58:43] does anyone have a snippet of code to format the rev_timestamp in a readable way (ie DD-MM-YY hh:mm) ? [20:59:22] What language, pleclown? [20:59:41] php [21:02:03] !log deployment-prep Converted deployment-sentry2.eqiad.wmflabs to use beta salt/puppet master [21:02:06] Logged the message, Master [21:07:40] <^d> pleclown: Either wfTimestamp() or Language's timeanddate() date() and time() functions for starters. [21:07:49] <^d> There's a bunch of date/time/etc functions in Language. [21:10:35] ^d: I'm not working on a mediawiki install. [21:11:21] <^d> In that case you probably want something like php's date() function. [21:12:33] I know, but as I'm lazy, I was hoping that someone else had already done the job in a tool, and was willing to share it with me :) [21:28:24] If someone is interested, the code is here https://github.com/Pleclown/Commons-Tools/blob/master/utils/functions.php (function FormatMWTimestamp) [22:28:02] Coren: around? question about NFS on non-tools project [22:35:14] YuviPanda: Does https://wikitech.wikimedia.org/wiki/Special:NovaInstance work for you? [22:35:24] I'm getting a crash page [22:35:26] bd808: roan broke wikitech, see -operations [22:35:34] ah [22:56:03] !log deployment-prep updated OCG to version aeb8623d6ebe41ae7c7e36c57844bd9ea8e6d595 [22:56:06] Logged the message, Master [23:01:44] Coren: btw, our graphite roles on trusty fail because python-txstatsd (our home built package) isn't available on trusty yet.