and accessdate? [19:21:26] <Dispenser> Next-gen reflinks would storage e.g. nytime.com/* then allow a human to easily write rules [19:21:54] <Cyberpower678> anomie, are you familiar with Flow's operation? [19:22:04] <anomie> Cyberpower678: Not really, why? [19:22:26] <Cyberpower678> anomie, I would like to know what table it writes to since it doesn't seem to write in revision. [19:22:28] <a930913> Dispenser: For a bot to later go through? [19:23:07] <Dispenser> Yes. It would greatly the quality of reference and likely solve any issues pertaining to references [19:23:21] <YuviPanda> andrewbogott: :) [19:23:23] <YuviPanda> andrewbogott: ok! [19:23:40] <anomie> Cyberpower678: I think it has it's own database. Hopefully still in MySQL, but I'm not even sure of that. [19:23:49] <YuviPanda> Dispenser: isn't that the job of archive.org? [19:23:59] <YuviPanda> or something like http://archive.today/ [19:24:33] <a930913> Dispenser: Sounds great. But instead of downloading the internet, why don't you fetch the pages as needed? [19:25:19] <Dispenser> YuviPanda: Remind me again why developers want SQL access and are unsatisfied with the REST API [19:25:49] <YuviPanda> Dispenser: I don't see how these are related. [19:26:10] <YuviPanda> but I see your mind is set and nothing I can say is going to change that, so I'll abstain from this conversation :) [19:26:38] <Cyberpower678> anomie, is it possible you could check real quick? [19:26:42] <Dispenser> a930913: The pattern recognition can only work if you have large dataset to pull from [19:28:22] <a930913> Dispenser: But one of the best things about the internet, is that you can fetch the pages on demand. [19:28:28] <Dispenser> 10 ms latency vs 2,000 ms latency [19:28:33] * anomie points Cyberpower678 to https://www.mediawiki.org/wiki/Flow_Portal/Team [19:28:38] <a930913> Dispenser: Yes. [19:28:48] <YuviPanda> Cyberpower678: you're better off asking #mediawiki-corefeatures [19:28:49] <Coren> Dispenser: Because there are queries possible to make when talking to the database that aren't when using the API? (Including exploiting relations between columns). [19:28:54] <a930913> Dispenser: $100 vs $10000 [19:29:32] <Cyberpower678> anomie, YuviPanda: Thanks and glad to see I'm not on your ignore list. :-) [19:29:38] <YuviPanda> Cyberpower678: :) [19:29:55] <Coren> Dispenser: And, again, I remind you that nobody told you no. Make a good case for it and present it to Engineering. It wouldn't be the first time we acquire/set aside resources for a project (to wit: OSM) [19:30:11] <Dispenser> WMF has enough money to make Wikipedia easier to read, use, and write. I'd suggest they use it wisely [19:30:58] <Cyberpower678> YuviPanda, unfortunately you linked me to an uninhabited channel. :p [19:31:03] <YuviPanda> Cyberpower678: ah, try #wikimedia-corefeatures? [19:31:39] <Coren> Dispenser: You mean for thinks like VE and Flow? :-) [19:31:44] <Coren> things* [19:32:47] <Cyberpower678> YuviPanda, Class Y planet. :p [19:33:19] <Dispenser> Coren: Reinventing the wheel is a waste of funds [19:33:23] <YuviPanda> Cyberpower678: aren't those uninhabited deserts? :) #wikimedia-corefeatures is useful [19:33:48] <Cyberpower678> YuviPanda, actually it's also known as a Demon class according to Star Trek. [19:34:01] <YuviPanda> Cyberpower678: indeed, and #wikimedia-corefeatures is class M [19:34:06] <YuviPanda> I just checked! [19:34:14] <YuviPanda> Cyberpower678: it even has a post-warp civilization! :) [19:34:34] <Coren> Dispenser: Indeed; no coincidence that Tool Labs requires open source tools so that the projects do not end up depending on software that ends up abandonned. [19:35:26] <Cyberpower678> YuviPanda, sensors confirm readings. [19:35:30] <Cyberpower678> :-) [19:35:32] * Dispenser <3 WMF rewrites of Open Source projects [19:35:33] <YuviPanda> Cyberpower678: :) [19:44:39] <andrewbogott> YuviPanda: btw, the wikitech vagrant role just leaves out SMW because I continue to be baffled/annoyed at the composer-only install path. [19:45:01] <andrewbogott> Someday some great hero will reconcile composer with vagrant and the other WMF install paths... [19:47:16] <a930913> DispenserAFK: Coren: In the meantime, can't reflinks be ported without the existing db, pending further discussion? [19:52:30] <DispenserAFK> a930913: I don't have to go to Labs. Could always build my own server and ask for donation. [19:52:40] <YuviPanda> andrewbogott: heh :) [19:53:45] <a930913> DispenserAFK: How soon can you do that? [19:56:09] <a930913> Coren: PM? [19:57:03] <Coren> a930913: Sure. [20:28:13] <a930913> DispenserAFK: How would you get 24TB to the labs btw? Drive it over? :p [20:32:24] <Hangsna> Hi! There is a bug if i do a search like this: https://tools.wmflabs.org/xtools/autoblock/?user=Ascilto&project=sv.wikipedia.org [20:32:42] <Hangsna> If i then click on the usernames i get to ENwp not SVwp in with i searched [20:41:25] <a930913> Hangsna: None of the developers listed are here online atm. [20:42:55] <Hangsna> a930913: okey, i could find somewere else for contact. Could you notify them for me? I guess its a rather quick fix if they just know about it. [20:43:37] <a930913> @notify Cyberpower678 [20:43:37] <wm-bot> I'll let you know when I see Cyberpower678 around here [20:44:15] <a930913> user181: What have you done with wm-bot?! [20:45:07] <a930913> Oh hold on, Cyberpower678 reads his memos... [20:47:26] <a930913> Hangsna: 21:46 -MemoServ(MemoServ@services.)- The memo has been successfully sent to Cyberpower678. [20:49:41] <Hangsna> thnx! [20:49:54] <Hangsna> exit [20:51:34] <a930913> Cyberpower678: Lol, talk about bad timing :p [21:08:27] <scfc_de> !log tools Reset queues in error state again [21:08:29] <labs-morebots> Logged the message, Master [21:10:24] <andrewbogott> labs-morebots, everything ok? [21:10:24] <labs-morebots> I am a logbot running on tools-exec-03. [21:10:24] <labs-morebots> Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [21:10:24] <labs-morebots> To log a message, type !log <msg>. [22:03:21] <akoopal> trusten [22:15:39] <wikibugs> 3Wikimedia Labs / 3tools: Make Flow database available / accessible on Labs/Tools - 10https://bugzilla.wikimedia.org/67397 (10metatron) 3UNCO p:3Unprio s:3normal a:3Marc A. Pelletier As mentioned in the docs, Flow is designed to be a cross wiki database https://www.mediawiki.org/wiki/Flow/Database P... [22:35:51] <wikibugs> 3Wikimedia Labs / 3deployment-prep (beta): Investigate broken puppet on Beta Cluster - 10https://bugzilla.wikimedia.org/67349#c1 (10Bryan Davis) Report on beta puppet run status: * ssh deployment-salt.eqiad.wmflabs * sudo salt '*' cmd.run '(hostname; date -d @$(grep last_run /var/lib/puppet/state/last_run_su... [22:53:34] <Betacommand> Coren: you around? [22:53:40] <Coren> Kinda [22:55:37] <a930913> We can't copy https://web.archive.org/web/20140324155321/http://toolserver.org/~dispenser/sources/ because DispenserAFK is holding the IP? [22:56:02] <Betacommand> never mind figured it out [22:56:14] <Betacommand> a930913: nope [22:56:43] <Coren> Let's just say that Dispenser's understanding of copyright and derived works is... unorthodox. [22:57:22] <wikibugs> 3Wikimedia Labs / 3deployment-prep (beta): Investigate broken puppet on Beta Cluster - 10https://bugzilla.wikimedia.org/67349#c2 (10Bryan Davis) Several of the failures reported above were caused by the local hacks that had been put in place to get puppet to run on deployment-apache0[12]. I have stashed thos... [22:57:32] <YuviPanda|zzz> a930913: not worth the trouble, IMO. [22:57:33] <jorm> do we need to talk to luis? [23:08:58] <legoktm> !log tools-pywikibot restarted webservice [23:08:58] <labs-morebots> tools-pywikibot is not a valid project. [23:09:15] <legoktm> !log tools tools-pywikibot started the webservice, don't know why it wasn't running [23:09:17] <labs-morebots> Logged the message, Master [23:21:07] <wikibugs> 3Wikimedia Labs / 3deployment-prep (beta): Investigate broken puppet on Beta Cluster - 10https://bugzilla.wikimedia.org/67349#c3 (10Bryan Davis) 5NEW>3RESO/FIX With the local hacks removed and a bad lock file deleted on deployment-jobrunner01, all hosts in beta are now reporting their latest puppet run a...

[00:06:16] I seem to be getting a disk quota error. [00:06:37] Carmela: ?? [00:06:56] Hmm, went away on retry. [00:07:16] Carmela: Im not sure labs has a quota in place [00:07:47] > cp: closing `mzmcbride@tools-login.wmflabs.org/home.tar.gz': Disk quota exceeded [00:07:53] Got a bunch of errors like that. [00:09:15] Carmela: is that the name of the tool? or your account? [00:09:41] It doesn't really matter. It's working now. [00:10:00] The Toolserver is dead. :-( [00:12:36] 3Wikimedia Labs / 3deployment-prep (beta): Yell loudly of failed puppet runs on Beta Cluster instances - 10https://bugzilla.wikimedia.org/67333#c4 (10Greg Grossmeier) (In reply to Yuvi Panda from comment #1) > Sadly we can't really use icinga properly on labs (so I'm told, due to the > way resource collectio... [00:20:51] Redirect 301 /~mzmcbride //tools.wmflabs.org/mzmcbride [01:09:51] 3Wikimedia Labs / 3deployment-prep (beta): Yell loudly of failed puppet runs on Beta Cluster instances - 10https://bugzilla.wikimedia.org/67333#c5 (10Yuvi Panda) (In reply to Greg Grossmeier from comment #4) > I wasn't thinking of necessarily copy/pasting the icinga config from prod, > but we have a beta la... [01:32:47] YuviPanda: /var is full on -exec-{03,10} again. The underlying problem (diamond logging too much) was mitigated previously by completely disabling diamond on Labs. If you enable it for Graphite, I think we need to cut down the amount of logging. [01:33:02] scfc_de: no, I think the solution is to enable biglogs [01:33:10] scfc_de: it's full on *all* of them? [01:33:13] oh dear. [01:33:42] Only on -03 and -10. [01:33:45] scfc_de: ah, ok [01:33:51] scfc_de: I'll clean them up and apply biglogs to them [01:34:02] YuviPanda: Already on that (the first part). [01:34:15] scfc_de: do you want to clean up, and I can apply biglogs? [01:34:47] !log tools tools-exec-03, tools-exec-10: Removed /var/log/diamond/diamond.log, restarted diamond and bzip2'ed /var/log/diamond/*.log.2014* [01:34:51] Logged the message, Master [01:35:00] YuviPanda: I'm still running bzip2, it should take a few more moments. [01:35:07] scfc_de: I usually just do rm diamond/diamond.log.* [01:35:10] and archive.log.* [01:35:15] scfc_de: I think that's safe enough [01:35:21] all these metrics are in graphite anyway [01:36:45] Yeah, that's why I think we could do without them :-). But biglogs is fine with me. [01:37:00] scfc_de: yeah, I'll make that change as well [01:37:03] but not today [01:37:08] it's already 7 AM :| [01:38:45] scfc_de: I'm restarting atop (what is it? why is it logging?), ssh and diamond after the move. [01:38:48] should be enough, I reckon [01:39:06] YuviPanda: As soon as bzip2, I'm off to bed as well :-). [01:39:13] *is finished [01:39:16] scfc_de: oh, you're still bzipping? I already moved it :| [01:40:10] Now bzip2 is finished on -exec-10. If previously you just rm'ed them, that's ... not really a problem :-). Okay, I'm off. Good night! [01:40:16] scfc_de: night! [01:40:30] !log tools tools-exec-03 applied biglogs, moved logs around, killed some old diamond logs [01:40:32] Logged the message, Master [01:41:23] !log tools tools-exec-03 restarted diamond, atop, exim4, ssh to pick up new log partition [01:41:25] Logged the message, Master [01:53:42] !log tools tools-exec-10 applied biglogs, moved logs around, killed some old diamond logs [01:53:45] Logged the message, Master [02:38:52] 3Wikimedia Labs / 3Infrastructure: Enable HTTPS on a Labs instance - 10https://bugzilla.wikimedia.org/54065#c11 (10scott.leea) What's the status on this ticket? Is this something I can work on? (I am a new volunteer). [02:59:36] 3Wikimedia Labs / 3deployment-prep (beta): Yell loudly of failed puppet runs on Beta Cluster instances - 10https://bugzilla.wikimedia.org/67333#c6 (10Greg Grossmeier) (In reply to Yuvi Panda from comment #5) > Theoretically, yeah :) But that effort was largely undocumented and > unpuppetized, and I don't kno... [03:11:36] 3Tool Labs tools / 3[other]: Migrate https://toolserver.org/~magnus/wpipe/pipeline.html - 10https://bugzilla.wikimedia.org/61176 (10This, that and the other) 5NEW>3RESO/WON [03:12:56] :'( [03:15:08] 3Tool Labs tools / 3[other]: Migrate https://toolserver.org/~magnus/flickr_mass.php - 10https://bugzilla.wikimedia.org/61174 (10This, that and the other) a:3Magnus Manske [03:36:46] (03CR) 10Jforrester: "Umm. LQT's maintainers are very definitely in Core Features, and it's their responsibilty…" [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/142991 (owner: 10Nemo bis) [03:38:53] (03PS1) 10Jforrester: Add some extra extensions and bug areas for the Editing team [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/143238 [03:42:15] (03PS2) 10Jforrester: Add some extra extensions and bug areas for the Editing team [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/143238 [03:42:17] (03PS1) 10Jforrester: Add Beta Labs bugs for the QA channel [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/143239 [03:43:59] (03CR) 10Greg Grossmeier: [C: 031] "+1, I would have submitted a slightly different style but same thing patch if he hasn't beat me to it." [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/143239 (owner: 10Jforrester) [03:56:04] Silke_afk: Ping me when you are around. [04:58:32] Good morning, Coren, ping! [04:59:18] What's the status of everything? [05:04:16] (03CR) 10Nemo bis: [C: 04-1] "Core must be followable on #wikimedia-dev." [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/143238 (owner: 10Jforrester) [05:06:46] (03CR) 10Nemo bis: Add some extra extensions and bug areas for the Editing team (031 comment) [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/143238 (owner: 10Jforrester) [05:10:46] (03CR) 10Legoktm: Add Beta Labs bugs for the QA channel (031 comment) [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/143239 (owner: 10Jforrester) [05:30:39] 3Wikimedia Labs / 3tools: Queue is mostly dead, which is slightly alive. - 10https://bugzilla.wikimedia.org/67347 (10bgwhite) 3UNCO p:3Unprio s:3major a:3Marc A. Pelletier 18 total queue slots are either in alarm or error state. One webserver slot is dead. As new queue jobs won't start on slots th... [06:50:25] 3Wikimedia Labs / 3deployment-prep (beta): Investigate broken puppet on Beta Cluster - 10https://bugzilla.wikimedia.org/67349 (10Greg Grossmeier) 3NEW p:3Unprio s:3normal a:3Ori Livneh Puppet on Beta Cluster broke sometime in the last week-ish; let's figure out where and fix it :) [06:50:36] 3Wikimedia Labs / 3deployment-prep (beta): Yell loudly of failed puppet runs on Beta Cluster instances - 10https://bugzilla.wikimedia.org/67333 (10Greg Grossmeier) [06:50:37] 3Wikimedia Labs / 3deployment-prep (beta): Investigate broken puppet on Beta Cluster - 10https://bugzilla.wikimedia.org/67349 (10Greg Grossmeier) p:5Unprio>3High [08:55:06] 3Wikimedia Labs / 3wikitech-interface: Enable HSTS (HTTP Strict Transport Security) on Wikitech - 10https://bugzilla.wikimedia.org/67303#c2 (10Jan Zerebecki) 5RESO/DUP>3REOP Wikitech can use a different, easier implementation compared to bug 38516 (wikipedia, etc.). As HTTPS can not be disabled in the pr... [09:06:53] 3Tool Labs tools / 3Erwin's tools: Migrate https://toolserver.org/~erwin85/blockfinder.php to Tool Labs - 10https://bugzilla.wikimedia.org/60880#c1 (10Andre Klapper) https://tools.wmflabs.org/erwin85/blockfinder.php still broken: Warning: There were MySQL errors at Tue, 01 Jul 2014 09:06:16 +0000. It is pos... [09:46:51] 3Wikimedia Labs / 3tools: Queue is mostly dead, which is slightly alive. - 10https://bugzilla.wikimedia.org/67347#c1 (10Andre Klapper) Commands to show that are very welcome, for curious non-techies like me. [10:05:21] 3Wikimedia Labs / 3tools: Queue is mostly dead, which is slightly alive. - 10https://bugzilla.wikimedia.org/67347#c2 (10Merlijn van Deen) $ qstat -f queuename qtype resv/used/tot. load_avg arch states -----------------------------------------------------------------------------... [10:05:51] 3Wikimedia Labs / 3tools: Queue is mostly dead, which is slightly alive. - 10https://bugzilla.wikimedia.org/67347#c3 (10Merlijn van Deen) and, from man qstat: If the state is a(larm) at least on of the load thresholds defined in the load_thresholds list of the queue configuration (see queue_conf(5))... [11:15:08] any admin around that could admin me to deployment-prep? I'd need to take a look for swift [11:18:46] godog: sure [11:19:16] hashar: thanks! [11:19:23] godog: what is your labs account? :D [11:20:04] ah https://wikitech.wikimedia.org/wiki/User:Filippo_Giunchedi [11:20:11] hashar: yep that! [11:20:31] !log deployment-prep Added Filippo Giunchedi to the project as an admin (WMF ops) [11:20:33] Logged the message, Master [11:20:52] granting sudo [11:21:14] godog: you should be all set [11:21:40] godog: andrewboggot faidon and I had a discussion last week about swift. Cant remember the details but basically ops is in charge of providing us a swift api entry point [11:21:53] I think it will be set up on top of labs instance in the deployment-prep project [11:22:52] godog: there is a local puppetmaster on deployment-salt.eqiad.wmflabs with a bunch of cherry picked patches. The local repo is rebased every hour or so via a cronjob in bd808 crontab [11:23:00] and we have salt! :D [11:24:24] hashar: nice! yeah I wanted to take a peek at how we could test the new swift version in beta (if at all) but I agree the medium-term goal would be to have swift in labs \cc andrewbogott_afk [11:24:43] hashar: anyway looks like I'm set, thanks! [11:47:01] !log tools tools-exec-05 was out of space in /var, removed old diamond logs and atop logs [11:47:06] !ping [11:47:06] !pong [11:49:36] anyone know how to reboot labs logsmorebot [11:52:33] YuviPanda: https://wikitech.wikimedia.org/wiki/Morebots ? [11:53:31] hmm, right [11:54:36] legoktm: I restarted [11:54:50] !log YuviPanda restarted something. [11:54:57] nothing yet [11:55:43] legoktm: wat, it just left [11:55:46] i'm so confused now [11:56:01] I think freenode is lagged [11:56:06] or my internet just sucks. [11:57:14] it hasn't been scheduled yet [11:57:14] odd [11:57:33] tools-exec-07 is now out of space [12:00:08] clear ALL the /vars \o/ [12:02:13] valhallasw: am also applying biglogs to them as I go, so this should be the last time (tm) /var gets filled [12:10:48] andrewbogott_afk: so, space issues for applying biglogs :| [12:10:49] andrewbogott_afk: Notice: /Stage[main]/Role::Labs::Lvm::Biglogs/Labs_lvm::Volume[logfile-disk]/Exec[create-vd-logfile-disk]/returns: Volume group "vd" has insufficient free space (1678 extents): 2048 required. [12:10:57] this was on tools-exec-05 [12:11:04] andrewbogott_afk: err, tools-exec-07 [12:12:05] andrewbogott_afk: I've removed the role for now [12:13:57] YuviPanda: What is this biglogs thing? [12:14:02] Anyone can restart wm-bot? [12:14:14] It became user181 :p [12:14:21] Silke_WMDE: allocates more space to /var/log so nodes don't run out of space because of big log files [12:14:31] Silke_WMDE: in the meantime I'm just clearing out old logs by hand which is a PITA [12:14:42] I've reduced the amount of logging our services do as well [12:14:43] i see! [12:37:06] 3Wikimedia Labs / 3tools: Queue is mostly dead, which is slightly alive. - 10https://bugzilla.wikimedia.org/67347#c4 (10metatron) Anyone willing to work on this? YuviPanda? Btw: queues + hostname with: $ qhost -q [12:40:06] 3Wikimedia Labs / 3tools: Some issues: tools-webgrid-03/04, tools-login - 10https://bugzilla.wikimedia.org/67329#c1 (10metatron) Puppet issues on webgri-02 + tools-login fixed now. Still not able to ssh into new nodes tools-webgrid-03/04 [13:29:44] !log deployment-prep rebuilding Cirrus search index in beta to pick up new configuration and cache warmers [13:29:46] Logged the message, Master [14:22:41] all queue seems frozen [14:46:16] andrewbogott: can I expect the proxy update to be deployed soon? [14:46:33] Krinkle: yes, as soon as there's a puppet run... [14:46:40] which might have happened already [14:47:03] Guess it needs a restar? [14:47:10] or soft reload of some kind [14:48:06] Yesterday I rerouted lots of tools to a catch-all tool-labs project as I haven't been able to migrate all tools yet. I can redirect them from there. One of the tools I took down for the moment due to this. Won'e be able to work on it for another while though. [14:48:26] example url that should return a custom page once ready: https://tools.wmflabs.org/krinkle-redirect/BlankPages.php404 [14:48:47] Ok, so this is the tools proxy that you're interested in? [14:49:15] I believe so [14:49:22] the main tools web server [14:50:52] Krinkle: try now? [14:50:59] great [14:51:00] thx [15:26:56] (03PS3) 10Jforrester: Add some extra extensions and bug areas for the Editing team [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/143238 [15:27:28] (03CR) 10Jforrester: "However, MediaWiki components are already defined in this list; if you want a wider fix, make a different patch." (031 comment) [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/143238 (owner: 10Jforrester) [15:27:38] (03PS2) 10Jforrester: Add Beta Labs bugs for the QA channel [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/143239 [15:28:30] (03CR) 10Jforrester: Add Beta Labs bugs for the QA channel (031 comment) [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/143239 (owner: 10Jforrester) [15:35:35] (03CR) 10Nemo bis: [C: 04-1] "I don't want a wider fix, I want "Page editing" to stay in the main channel." [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/143238 (owner: 10Jforrester) [15:36:45] (03CR) 10Jforrester: "Please justify what makes one of the least-active components of MediaWiki, with the fewest bugs, more important than Uploading against whi" [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/143238 (owner: 10Jforrester) [15:36:58] 3Wikimedia Labs / 3tools: Queue is mostly dead, which is slightly alive. - 10https://bugzilla.wikimedia.org/67347#c5 (10Tim Landscheidt) a:5Marc A. Pelletier>3Yuvi Panda I've reset all queues, but they become stuck again because /var is full on the hosts. Assigning to Yuvi for diamond & biglog. [15:38:05] Coren: lvm seems to have run out of space [15:38:15] o_O? [15:38:18] Coren: Notice: /Stage[main]/Role::Labs::Lvm::Biglogs/Labs_lvm::Volume[logfile-disk]/Exec[create-vd-logfile-disk]/returns: Volume group "vd" has insufficient free space (1678 extents): 2048 required. [15:38:23] and puppet fails [15:38:41] Which host is that on? [15:39:50] Coren: exec-07 [15:40:26] Coren: I removed the role to let puppet run [15:42:23] -07, for some reason, has a smaller disk that the others. [15:42:34] Coren: O_o [15:43:05] ... it's the wrong size instance. [15:43:17] D'oh! [15:45:13] Coren: hah. I think we should delete and re-create it. [15:45:36] Coren: a patch got merged yesterday that keeps less diamond logs (only 1 day, vs 5 now) [15:46:43] !log deployment-prep Fixed git rebase conflict in operations/puppet on deployment-salt [15:46:45] Logged the message, Master [15:46:47] YuviPanda: That's nontrivial (because key exchange, etc). At the very least I'm going to reduce its allocation slice in gridengine and create the logs lvm by hand though. [15:47:03] Coren: ah, ok. [15:49:15] Coren: can you log when you create the log lvm volume? [15:50:02] !log tools created logfile disk for -exec-07 by hand (smaller instance) [15:50:05] Logged the message, Master [15:51:05] !log tools adjusted resource limits for -exec-07 to match the smaller instance size. [15:51:07] Logged the message, Master [15:51:57] Coren: I should setup a dsh or salt thing that lets me run commands on clusters of these things [15:53:47] Coren: fixing tools-exec-02 now [15:54:02] I have some naive scripts in my home for those purposes, but needing manual maintenance. Salt would be +good. [15:54:43] !log tools tools-exec-02 removed stale puppet pid file, forcing run [15:54:44] Logged the message, Master [15:55:02] Coren: yeah, definitely [15:55:19] Coren: I think I'll have to spend some time doing $DAYJOB specific things rest of the week, though :( [15:56:04] YuviPanda: A word to the wise; should you ever end up ops-side, you're likely to be often grabbed for labsish stuff given how familiar you've gotten with the setup. :-P [15:56:24] Congrats on the transition all [15:56:36] Coren: also wanted to ask, is it possible to get a disk image with local disk > 160GB? graphite will run out of that in some time (weeks?) if we don't [15:56:58] Coren: :D Will keep in mind [15:57:17] sumanah: \o/. I was reading through old toolserver-l archives from Octoberish 2013, things seem much better now :) [15:58:51] 3Wikimedia Labs / 3Infrastructure: Enable HTTPS on a Labs instance - 10https://bugzilla.wikimedia.org/54065#c12 (10Yuvi Panda) 5UNCO>3RESO/FIX This has been fixed now. If you are using your own instance, use Special:NovaProxy on wikitech to create a https enabled proxy. If you're using toollabs, you alre... [15:59:09] YuviPanda: Not from the current image set, no. That said, there's nothing that prevents the graphite instance from rotating logs away from the local disk unto project storage. [15:59:30] sumanah: Thanks. [15:59:47] Coren: ah, hmm. right, but that'd complicate things, I think. Perhaps we should just reduce granurality over time, and also make sure that we don't overlog things. [15:59:50] speaking of which... [16:00:21] Coren: Hi. Is there any update on the database table issue I noted recently? [16:00:22] 3Wikimedia Labs / 3deployment-prep (beta): Setup monitoring for Beta cluster - 10https://bugzilla.wikimedia.org/51497#c5 (10scott.leea) If this is still an issue can I work on it? If so, please provide any additional details I can to get started. [16:03:02] krd: I've been working pretty much nonstop on it but I may have run into an upstream bug with the federation engine. In the meantime, if your application allows it, the issue is not present if you use commons directly as opposed through federation. [16:05:02] ok [16:07:52] !log tools applied biglogs to tools-exec-02 and rejigged things [16:07:54] Logged the message, Master [16:45:21] (03CR) 10Merlijn van Deen: [C: 032 V: 032] Restrict ContentTranslation to #mediawiki-i18n [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/142992 (owner: 10Nemo bis) [16:47:10] !log restarted wikibugs with new channel config / https://gerrit.wikimedia.org/r/#/c/142992/ / Nemo_bis [16:47:10] restarted is not a valid project. [16:47:16] !log wikibugs restarted wikibugs with new channel config / https://gerrit.wikimedia.org/r/#/c/142992/ / Nemo_bis [16:47:17] Logged the message, Master [16:52:25] Coren: I've to stop doing things for a while because of my CTS, don't think I'll get to biglogging / clearing out /var/log on the other instances. Can you do that to make sure the queues are running? [16:52:50] I will, intermittently over the next couple days. [16:53:06] My #1 thing is a couple of blockers db-side [16:53:30] Coren: alright. Just wanted to make sure that the queues were working [16:53:35] Coren, load is badly balanced on exec nodes, some hosts at 25% cpu some other at 200% load [16:53:51] 'cause their sge queue is frozen [16:54:18] phe: Yes, as the logdirs get cleaned they'll come back into rotation. [17:01:21] 3Wikimedia Labs / 3deployment-prep (beta): Automate updating the puppet checkout - 10https://bugzilla.wikimedia.org/66683#c5 (10Bryan Davis) Created attachment 15801 --> https://bugzilla.wikimedia.org/attachment.cgi?id=15801&action=edit bash script to update puppet checkout in /var/lib/git/operations/puppe... [17:51:55] !log tools tools-exec-04 removed stale pid file and force puppet run [17:51:57] Logged the message, Master [18:10:18] Coren: you around? [18:10:41] Betacommand: I see all, and hear all. [18:10:53] But no, I'm a human. :-) [18:10:55] Coren: then take a look at ANI [18:11:25] The bit about Dispenser? I've already commented. Something new came up? [18:11:48] Waiting 24 hours [18:12:20] Ah, no, ANI not AN. /me goes look. [18:12:21] Coren: No, about mass sock issues [18:12:39] begging for a CU [18:17:08] Betacommand: The thread about archive.is? [18:17:23] yeah, looks like a massive sock farm [18:20:44] DQ already looked at it; using proxies. This needs and edit filter. [18:22:10] Coren: thanks [18:23:32] Dispenser: While you're here, would you care to elaborate on why you need 24T? If you give a bit more background, it becomes considerably easier to see how things can be adapted. [18:24:15] First the average size of web page is 1 MB these day (includes CSS+Images+JS+Video) [18:24:47] Times 24 million links for English Wikipedia is 24 TB [18:25:16] Hopefully I can compress to include other Wikipedias and future growth [18:28:01] I'd also like to extra text and previous/follow pages of a multi-page article [18:28:22] As possibly compare original to future updates [18:30:57] ... wait, you're actually archiving the *target* of the link? [18:31:09] wouldn't that have copyright issues? [18:31:09] Yes [18:31:27] It does have copyright issues; also, are/were you doing that on toolserver? [18:31:28] We're in the US [18:31:33] No [18:31:40] err, copyright is only relevant when you redistribute the contents [18:32:38] valhallasw: No it's not. To wit: according to US case law the copy in ram of a program from a disk is "a copy" and subject, hence the insanity about being allowed to just license the software for limited uses. [18:33:10] valhallasw: it'd be possible to mount a reasonable fair use case in the case of archival/academic use, but it's a complicated legal minefield. [18:33:38] Hum, okay. [18:33:39] Unable to run job: error writing object "1994948" to spooling database [18:33:39] aborting transaction (rollback) [18:33:39] (see also: Google Books, Google News) [18:33:39] job 1994948 was rejected cause it couldn't be written. [18:33:53] ^ that's not suppposed to happen, right? [18:34:26] Dispenser: allright, so my previous comment about needing an okay from WMF engineering for the resources stands -- but how exactly is it preventing you from porting the tools that were running on toolserver? [18:35:04] Why should I port obsolete tools? [18:35:12] valhallasw: No it's not. Lemme see if diamond logs are the cause. [18:35:40] Dispenser: Well, I'd argue that any tool that is in active use is, in fact, not obsolete by definition. :-) [18:36:09] You could say the same about the Toolserver :-) [18:37:53] It's pretty clear that the toolserver was not obsolete for that very reason. It certainly was obsolescent and in need of being replaced, which it has. [18:38:29] And we had a TS replacement before the TS was removed [18:38:40] Used planned Linux back migration [18:39:02] Dispenser: If you don't *want* to port your tools because you don't want to bother, or as a statement about/against the move from ts to labs, it's perfectly within your rights -- but please don't say that this is because the WMF doesn't want you to or refuses to give you the resources to do so. [18:40:14] The fact that we won't allocate tens of thousands of dollars of resources for a future tool you are planning without a solid use case and plan is not unreasonable. [18:41:03] valhallasw: Yeah, it was diamond filling a filesystem again. Should be fixed. [18:41:48] Coren: thanks [19:03:27] Coren: Did the cluebot webservice get caught in that webgrid problem? [19:03:46] a930913: It's possible. Lemme go give it a swift kick in the diodes. [19:04:20] (Do we know where Damianz is?) [19:04:43] I need to find a clean way by which tools can specify whether their tools is safe to restart automatically; that'd make things much more robust because I could automate it. [19:05:05] a930913: I've been poking him for a while as well, no response [19:05:20] YuviPanda: It's been what? At least two months now? [19:05:37] Coren, tools.supercount@tools-login:~$ mysql enwiki [19:05:37] ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) [19:05:51] Cyberpower678: ... yes? [19:06:04] Cyberpower678: That seems quite normal. There is no mysqld on the login host. [19:06:14] Well. I can't access the DB on the terminal [19:06:17] Cyberpower678: $ sql enwiki [19:06:45] Oh. Yeah, as a930913 just said, you probably wanted to 'sql' helper script, not mysql directly. :-) [19:06:47] a930913, oh right. Thanks. :p [19:07:20] * a930913 hands Cyberpower678 a trout. [19:07:44] * Cyberpower678 accepts it and eats it. [19:09:37] * a930913 waits for the explosive to go off. [19:10:26] * Cyberpower678 reminds a930913 that he is a plasma entity and that explosives have no effect. :D [19:11:10] Dispenser: You were downloading the internet?! [19:11:30] Cyberpower678: Oh, you should meet my nanobot friend. [19:11:52] It would vaporize if it touched me. [19:11:56] I'm hot. [19:11:59] :p [19:12:12] Cyberpower678: He just reassembles himself. [19:12:43] a930913, how could he reassemble himself if he's been vaporized/ [19:12:51] a930913: Imagine if a bot could check the factual accuracy of enwiki. Its in the realm of possibilities [19:13:24] Cyberpower678: Iridium or something. [19:13:52] Dispenser: Sure, but not with current technology. "Dump now, parse later"? [19:13:58] * Cyberpower678 uses that to buy himself a backpack upgrade? [19:15:16] Soon than you think cuz' of the /way/ Wikipedia is written [19:17:11] Dispenser: Can I see the code? I keep asking people to explain to me what reflinks was/did, but nobody will :( [19:17:52] andre__: any luck with that wikitech vagrant role? [19:18:08] You gave it a URL, it download the page and filled out a citation for you [19:18:16] YuviPanda: did you really mean me? :) [19:18:25] andre__: gah, sorry. autocomplete fail. [19:18:34] phou... I was hoping for that :D [19:18:36] andrewbogott: any luck with the wikitech vagrant role? [19:18:41] andre__: :D [19:19:08] YuviPanda: yeah, it sort of works. [19:19:16] Requires a ton of hand tuning to make it actually talk to openstack and ldap [19:19:19] andrewbogott: woo! did that get merged? [19:19:27] andrewbogott: ah, does it talk to prod openstack/ldap? [19:19:30] but it's working well enough that I can test what i wanted to test :) [19:19:34] No, a local install [19:19:42] andrewbogott: ah, cool! :D Submit? :) [19:19:43] so it's a weird hybrid of prod puppet classes and vagrant stuff [19:19:45] as a patch, I mean [19:19:57] Ori merged the vagrant bits. there are a few other parts that still need review [19:20:06] aaah, cool! [19:20:06] Dispenser: Excepting the saving of the page, why would it be resource intensive? [19:20:59] YuviPanda: for now I have deferred my dream of having a fully working wikitech come up like magic from puppet. [19:21:03] Although I will work on it more sometime. [19:21:04] Dispenser: Url, and accessdate? [19:21:26] <Dispenser> Next-gen reflinks would storage e.g. nytime.com/* then allow a human to easily write rules [19:21:54] <Cyberpower678> anomie, are you familiar with Flow's operation? [19:22:04] <anomie> Cyberpower678: Not really, why? [19:22:26] <Cyberpower678> anomie, I would like to know what table it writes to since it doesn't seem to write in revision. [19:22:28] <a930913> Dispenser: For a bot to later go through? [19:23:07] <Dispenser> Yes. It would greatly the quality of reference and likely solve any issues pertaining to references [19:23:21] <YuviPanda> andrewbogott: :) [19:23:23] <YuviPanda> andrewbogott: ok! [19:23:40] <anomie> Cyberpower678: I think it has it's own database. Hopefully still in MySQL, but I'm not even sure of that. [19:23:49] <YuviPanda> Dispenser: isn't that the job of archive.org? [19:23:59] <YuviPanda> or something like http://archive.today/ [19:24:33] <a930913> Dispenser: Sounds great. But instead of downloading the internet, why don't you fetch the pages as needed? [19:25:19] <Dispenser> YuviPanda: Remind me again why developers want SQL access and are unsatisfied with the REST API [19:25:49] <YuviPanda> Dispenser: I don't see how these are related. [19:26:10] <YuviPanda> but I see your mind is set and nothing I can say is going to change that, so I'll abstain from this conversation :) [19:26:38] <Cyberpower678> anomie, is it possible you could check real quick? [19:26:42] <Dispenser> a930913: The pattern recognition can only work if you have large dataset to pull from [19:28:22] <a930913> Dispenser: But one of the best things about the internet, is that you can fetch the pages on demand. [19:28:28] <Dispenser> 10 ms latency vs 2,000 ms latency [19:28:33] * anomie points Cyberpower678 to https://www.mediawiki.org/wiki/Flow_Portal/Team [19:28:38] <a930913> Dispenser: Yes. [19:28:48] <YuviPanda> Cyberpower678: you're better off asking #mediawiki-corefeatures [19:28:49] <Coren> Dispenser: Because there are queries possible to make when talking to the database that aren't when using the API? (Including exploiting relations between columns). [19:28:54] <a930913> Dispenser: $100 vs $10000 [19:29:32] <Cyberpower678> anomie, YuviPanda: Thanks and glad to see I'm not on your ignore list. :-) [19:29:38] <YuviPanda> Cyberpower678: :) [19:29:55] <Coren> Dispenser: And, again, I remind you that nobody told you no. Make a good case for it and present it to Engineering. It wouldn't be the first time we acquire/set aside resources for a project (to wit: OSM) [19:30:11] <Dispenser> WMF has enough money to make Wikipedia easier to read, use, and write. I'd suggest they use it wisely [19:30:58] <Cyberpower678> YuviPanda, unfortunately you linked me to an uninhabited channel. :p [19:31:03] <YuviPanda> Cyberpower678: ah, try #wikimedia-corefeatures? [19:31:39] <Coren> Dispenser: You mean for thinks like VE and Flow? :-) [19:31:44] <Coren> things* [19:32:47] <Cyberpower678> YuviPanda, Class Y planet. :p [19:33:19] <Dispenser> Coren: Reinventing the wheel is a waste of funds [19:33:23] <YuviPanda> Cyberpower678: aren't those uninhabited deserts? :) #wikimedia-corefeatures is useful [19:33:48] <Cyberpower678> YuviPanda, actually it's also known as a Demon class according to Star Trek. [19:34:01] <YuviPanda> Cyberpower678: indeed, and #wikimedia-corefeatures is class M [19:34:06] <YuviPanda> I just checked! [19:34:14] <YuviPanda> Cyberpower678: it even has a post-warp civilization! :) [19:34:34] <Coren> Dispenser: Indeed; no coincidence that Tool Labs requires open source tools so that the projects do not end up depending on software that ends up abandonned. [19:35:26] <Cyberpower678> YuviPanda, sensors confirm readings. [19:35:30] <Cyberpower678> :-) [19:35:32] * Dispenser <3 WMF rewrites of Open Source projects [19:35:33] <YuviPanda> Cyberpower678: :) [19:44:39] <andrewbogott> YuviPanda: btw, the wikitech vagrant role just leaves out SMW because I continue to be baffled/annoyed at the composer-only install path. [19:45:01] <andrewbogott> Someday some great hero will reconcile composer with vagrant and the other WMF install paths... [19:47:16] <a930913> DispenserAFK: Coren: In the meantime, can't reflinks be ported without the existing db, pending further discussion? [19:52:30] <DispenserAFK> a930913: I don't have to go to Labs. Could always build my own server and ask for donation. [19:52:40] <YuviPanda> andrewbogott: heh :) [19:53:45] <a930913> DispenserAFK: How soon can you do that? [19:56:09] <a930913> Coren: PM? [19:57:03] <Coren> a930913: Sure. [20:28:13] <a930913> DispenserAFK: How would you get 24TB to the labs btw? Drive it over? :p [20:32:24] <Hangsna> Hi! There is a bug if i do a search like this: https://tools.wmflabs.org/xtools/autoblock/?user=Ascilto&project=sv.wikipedia.org [20:32:42] <Hangsna> If i then click on the usernames i get to ENwp not SVwp in with i searched [20:41:25] <a930913> Hangsna: None of the developers listed are here online atm. [20:42:55] <Hangsna> a930913: okey, i could find somewere else for contact. Could you notify them for me? I guess its a rather quick fix if they just know about it. [20:43:37] <a930913> @notify Cyberpower678 [20:43:37] <wm-bot> I'll let you know when I see Cyberpower678 around here [20:44:15] <a930913> user181: What have you done with wm-bot?! [20:45:07] <a930913> Oh hold on, Cyberpower678 reads his memos... [20:47:26] <a930913> Hangsna: 21:46 -MemoServ(MemoServ@services.)- The memo has been successfully sent to Cyberpower678. [20:49:41] <Hangsna> thnx! [20:49:54] <Hangsna> exit [20:51:34] <a930913> Cyberpower678: Lol, talk about bad timing :p [21:08:27] <scfc_de> !log tools Reset queues in error state again [21:08:29] <labs-morebots> Logged the message, Master [21:10:24] <andrewbogott> labs-morebots, everything ok? [21:10:24] <labs-morebots> I am a logbot running on tools-exec-03. [21:10:24] <labs-morebots> Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [21:10:24] <labs-morebots> To log a message, type !log <msg>. [22:03:21] <akoopal> trusten [22:15:39] <wikibugs> 3Wikimedia Labs / 3tools: Make Flow database available / accessible on Labs/Tools - 10https://bugzilla.wikimedia.org/67397 (10metatron) 3UNCO p:3Unprio s:3normal a:3Marc A. Pelletier As mentioned in the docs, Flow is designed to be a cross wiki database https://www.mediawiki.org/wiki/Flow/Database P... [22:35:51] <wikibugs> 3Wikimedia Labs / 3deployment-prep (beta): Investigate broken puppet on Beta Cluster - 10https://bugzilla.wikimedia.org/67349#c1 (10Bryan Davis) Report on beta puppet run status: * ssh deployment-salt.eqiad.wmflabs * sudo salt '*' cmd.run '(hostname; date -d @$(grep last_run /var/lib/puppet/state/last_run_su... [22:53:34] <Betacommand> Coren: you around? [22:53:40] <Coren> Kinda [22:55:37] <a930913> We can't copy https://web.archive.org/web/20140324155321/http://toolserver.org/~dispenser/sources/ because DispenserAFK is holding the IP? [22:56:02] <Betacommand> never mind figured it out [22:56:14] <Betacommand> a930913: nope [22:56:43] <Coren> Let's just say that Dispenser's understanding of copyright and derived works is... unorthodox. [22:57:22] <wikibugs> 3Wikimedia Labs / 3deployment-prep (beta): Investigate broken puppet on Beta Cluster - 10https://bugzilla.wikimedia.org/67349#c2 (10Bryan Davis) Several of the failures reported above were caused by the local hacks that had been put in place to get puppet to run on deployment-apache0[12]. I have stashed thos... [22:57:32] <YuviPanda|zzz> a930913: not worth the trouble, IMO. [22:57:33] <jorm> do we need to talk to luis? [23:08:58] <legoktm> !log tools-pywikibot restarted webservice [23:08:58] <labs-morebots> tools-pywikibot is not a valid project. [23:09:15] <legoktm> !log tools tools-pywikibot started the webservice, don't know why it wasn't running [23:09:17] <labs-morebots> Logged the message, Master [23:21:07] <wikibugs> 3Wikimedia Labs / 3deployment-prep (beta): Investigate broken puppet on Beta Cluster - 10https://bugzilla.wikimedia.org/67349#c3 (10Bryan Davis) 5NEW>3RESO/FIX With the local hacks removed and a bad lock file deleted on deployment-jobrunner01, all hosts in beta are now reporting their latest puppet run a...