[00:26:27] 06Labs, 10Tool-Labs: Mail doesn't work for nested tool accounts - https://phabricator.wikimedia.org/T65163#3063206 (10scfc) (My changes https://gerrit.wikimedia.org/r/#/c/237871 and https://gerrit.wikimedia.org/r/#/c/148917/ would make this possible.) [01:15:59] (03PS2) 10Tim Landscheidt: Switch list.php to proxymanager's new API [labs/toollabs] - 10https://gerrit.wikimedia.org/r/268343 [01:19:17] (03PS2) 10Tim Landscheidt: WIP: Don't ignore fchdir()'s errors [labs/toollabs] - 10https://gerrit.wikimedia.org/r/331227 [01:20:36] (03CR) 10jerkins-bot: [V: 04-1] WIP: Don't ignore fchdir()'s errors [labs/toollabs] - 10https://gerrit.wikimedia.org/r/331227 (owner: 10Tim Landscheidt) [01:29:09] 10Tool-Labs-tools-Other, 06Wikisource: OAuth time out is too short - https://phabricator.wikimedia.org/T135279#3063370 (10Samwilson) 05Open>03Resolved a:03Samwilson I've increased the cookie lifetime to 30 days. [01:41:13] 06Labs, 10Tool-Labs: Puppetize gridengine master configuration - https://phabricator.wikimedia.org/T95747#3063408 (10scfc) @gerritbot does not seem to want to add https://gerrit.wikimedia.org/r/#/c/230477/ to this task; hmmm. Anyway, that patch only handles the part of the grid master configuration dealt with... [01:53:08] 10Labs-project-Wikistats: miraheze custom domains not updated on wikistats - https://phabricator.wikimedia.org/T158976#3063438 (10Dzahn) Is it possible that there was a change on your side that means now custom wikis are reported in BOTH, the regular list AND the custom list? It seems like that is the root caus... [01:54:16] 06Labs, 10Tool-Labs, 13Patch-For-Review: Make role::aptly::server work with Apache - https://phabricator.wikimedia.org/T153814#3063444 (10scfc) (CCing @mmodell because of 5baa9cedd427110ab84b2cb4e0b24f65cbfa6986.) [01:57:12] 06Labs, 10Tool-Labs, 13Patch-For-Review: Make role::aptly::server work with Apache - https://phabricator.wikimedia.org/T153814#3063452 (10scfc) a:05scfc>03None [02:29:50] (03PS109) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [02:44:26] (03CR) 10Ricordisamoa: [C: 04-2] "PS109 uses new-style string entity id serialization instead of prefix + numeric id" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [04:49:34] 06Labs, 10Tool-Labs, 13Patch-For-Review, 07Puppet: Make standalone puppetmasters optionally use PuppetDB - https://phabricator.wikimedia.org/T153577#3063602 (10scfc) I'm pretty sure the patches work except that I can't get them to work on `toolsbeta-puppetmaster7` due to some PostgreSQL hiccups (our puppet... [04:49:48] 06Labs, 10Tool-Labs, 13Patch-For-Review, 07Puppet: Make standalone puppetmasters optionally use PuppetDB - https://phabricator.wikimedia.org/T153577#3063603 (10scfc) (… or anyone else can do that.) [06:40:39] PROBLEM - Puppet run on tools-exec-1406 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:45:50] PROBLEM - Puppet run on tools-webgrid-lighttpd-1409 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [07:20:39] RECOVERY - Puppet run on tools-exec-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [07:25:52] RECOVERY - Puppet run on tools-webgrid-lighttpd-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [08:16:44] 06Labs: Request creation of GLAMpipe labs project - https://phabricator.wikimedia.org/T159309#3063738 (10Zache) [09:51:59] 06Labs, 10Tool-Labs, 10Tools-Kubernetes, 13Patch-For-Review: k8s webservice restart failure with `ValueError: get() more than one object; use filter` - https://phabricator.wikimedia.org/T156626#3063953 (10zhuyifei1999) p:05High>03Normal @JustBerry Do you have the steps in which you produced the issue?... [10:57:48] 06Labs, 10WM-Bot: Move wm-bot instance to Trusty - https://phabricator.wikimedia.org/T157838#3064063 (10Petrb) Hello, yes this is related to both issues, but wm-bot is still logging all channels, just the data are now stored somewhere else, I need to move the web services and merge the log files, should be don... [11:09:17] 06Labs, 10WM-Bot: Move wm-bot instance to Trusty - https://phabricator.wikimedia.org/T157838#3064103 (10Petrb) Hi, browser is back, also please don't use old "bots.wmflabs.org" domain, it was deprecated some time ago, correct one is wm-bot.wmflabs.org I can't guarantee there will be no more outages, 2 more in... [11:15:29] I'm trying to set up my own puppetmaster and agents on labs following https://wikitech.wikimedia.org/wiki/Standalone_puppetmaster. Although I seem to have a working master and client on the same host when I try to spawn a new client it never seems to push my ssh keys so I can log in. The horizon console shows "puppet-agent[554]: Could not request [11:15:29] certificate: getaddrinfo: Name or service not known". This client is a totally vanilla jessie small instance. [12:54:08] tarrow: usually that is a DNS issue [12:54:27] can you ssh to the instance? [12:54:47] ok, I've spawned a new instance and it hasn't repeated [12:54:56] you also need a hiera configuration variable: puppetmaster: name.of.your.puppetmater.here [12:54:57] hey :} [12:55:16] I could make a connection to 22 but not login (no key) [12:56:39] Now I'm just struggling to make my new client succeed in connecting to the puppetmaster. Should I need to add a security group for that? [12:57:26] are you able to ssh to the instance? [12:58:27] tarrow: there is also a bug that causes puppet to self on new instance when using puppet standalone [12:58:28] :( [12:59:34] tarrow: maybe you are hit by https://phabricator.wikimedia.org/T148929 "New instance have broken puppet configuration when using puppetmaster standalone" [12:59:39] that happens on jessie [12:59:50] and yields something like: arning: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: deployment-puppetmaster.deployment-prep.eqiad.wmflabs] [12:59:52] I'll take a look :) [13:00:07] you can force run using: sudo puppet agent -tv [13:00:15] or look at /var/log/puppet/puppet.log [13:00:20] or maybe that is /var/log/puppet.log [13:00:22] "Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': Connection timed out - connect(2) for "puppetmaster-06.wikifactmine.eqiad.wmflabs" port 8140" [13:00:27] ah [13:00:31] so that is different [13:00:36] from a very slow forced run [13:00:38] 8140 I have no idea what it is used for [13:00:56] might want to verify that puppet runs fine on your puppetmaster-06 instance [13:01:36] and possibly have to restart apache on puppetmaster-06 [13:01:50] ok [13:02:24] on the puppetmaster you can verify it has a process listening on port 8140 using: [13:02:26] $ sudo netstat -tlnp|grep :8140 [13:02:31] tcp6 0 0 :::8140 :::* LISTEN 17108/apache2 [13:02:56] (netstat options: -l list , -t TCP , -n do not resolve names, -p show the process name (needs root) [13:03:15] so potentially you can : sudo systemctl restart apache2 [13:03:26] (assuming the puppetmaster is Jessie ) [13:04:07] I've basically managed to get myself in a right pickle (hence I'm already up to puppetmaster-06) but I'm finding that the problems I hit seem to be different on each "fresh" start [13:05:00] is it acceptable to have a Hiera "puppetmaster: fqdn" for the whole project set on wikitech? [13:10:28] tarrow: probably [13:10:36] that would cause all your instance to point to that puppetmaster [13:11:05] via I guess https://horizon.wikimedia.org/project/puppet/ [13:11:41] I am not entirely sure how it works to be honest [13:12:12] cool; so that is how I had it set up [13:12:47] but using the wikitech Hiera wikipages as the place to put configs [13:13:22] I'm fairly certain you I'm hitting T148929 [13:13:22] T148929: New instance have broken puppet configuration when using puppetmaster standalone - https://phabricator.wikimedia.org/T148929 [13:18:00] tarrow: can you paste the output of a run of puppet agent -tv on your instance? [13:18:08] https://phabricator.wikimedia.org/paste/ :} [13:19:01] sure; I also seem to have it broken on the puppetmaster as well :/ I'll post from there first [13:21:02] https://phabricator.wikimedia.org/P5007 [13:21:32] (obviously I've followed the instructions in the output) [13:28:49] tarrow: could you be caught up in this? https://phabricator.wikimedia.org/T159254 [13:32:14] probably if you follow the given instructions that might resolve it [13:32:37] I did think about this but it doesn't seem so, apache is up to date and I've restarted [13:56:30] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3064388 (10chasemp) ```We need to take labsdb1006.eqiad.wmnet and labsdb1007.eqiad.wmnet offline to update them from Ubuntu Precise to Debian Jessie on 2017-03-08. This... [13:56:43] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3064389 (10chasemp) [14:34:34] tarrow: I'm sorry we are seriously short handed today, I'll attempt to get T148929 looked at when I can [14:34:35] T148929: New instance have broken puppet configuration when using puppetmaster standalone - https://phabricator.wikimedia.org/T148929 [14:36:01] chasemp: no worries. I'm currently proving so puppet incompetent that I'll be happy when I just find repeatable failure let alone success... [14:36:12] :) [15:03:34] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Documentation: Create a "my first Pywikibot bot" tutorial for Tool Labs - https://phabricator.wikimedia.org/T134495#3064564 (10Aklapper) @Xephyr826: Thanks! Please tell us if you have any specific questions or get stuck so we can provide input! [15:40:32] chasemp: can you take a look at job 1820616, please? it doesn't schedule on tools-exec-gift-trusty-01.tools.eqiad.wmflabs and i don't exactly see why [15:42:09] annika: I will when I get a moment, we are really short handed today [15:42:22] that's fine, ty :) [16:32:37] 10Striker: Fix repo in toolsadmin after being renamed in Phab - https://phabricator.wikimedia.org/T159343#3064806 (10MarcoAurelio) [16:34:28] 10Striker, 10Tool-Labs-tools-stewardbots: Link existing repo in Toolsadmin - https://phabricator.wikimedia.org/T159344#3064825 (10MarcoAurelio) [16:34:46] 10Striker, 10Tool-Labs-tools-Quentinv57's-tools: Fix repo in toolsadmin after being renamed in Phab - https://phabricator.wikimedia.org/T159343#3064838 (10MarcoAurelio) [16:38:06] (03CR) 10Lokal Profil: "So the difference is that tracked photos are the subset which didn't trigger any of the exceptions?" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/338009 (owner: 10Jean-Frédéric) [17:06:35] I have a problem with CORS on Chrome which i did not have a week ago and I'm wondering if it's because some configurations changed in http://tools.wmflabs.org/. I make a request to http://etytree-virtuoso.wmflabs.org/sparql from javascript http://tools.wmflabs.org/etytree/etymology/resources/js/query.js [17:06:55] but I get this error on Chrome: XMLHttpRequest cannot load https://etytree-virtuoso.wmflabs.org/sparql?query=PREFIX%20dbnary%3A%20%3Ch…ND(if%20(%3Fll%20%3D%20%22%22%2C%22eng%22%2C%3Fll)%20AS%20%3Fiso%20)%20%7D. The 'Access-Control-Allow-Origin' header contains multiple values 'http://tools.wmflabs.org, *', but only one is allowed. Origin 'http://tools.wmflabs.org' is therefore not allowed access. [17:07:24] everything works on firefox [17:07:28] any ideas? [17:14:58] 06Labs, 10WM-Bot: Move wm-bot instance to Trusty - https://phabricator.wikimedia.org/T157838#3064950 (10Niedzielski) Thanks, I've asked for someone with channel permissions to update the URL to: https://wm-bot.wmflabs.org/logs/%23wikimedia-mobile/ [17:14:58] Hey wikibugs_, you are welcome! [17:19:06] wm-bot: ... [17:19:07] Hi zhuyifei1999_, there is an error, I'm a stupid bot and not intelligent enough to hold a conversation with you :-) [17:20:05] Epantaleo: : not really chome is fine for me there, but if you want to pursue a task w/ specific versions of everything is the best thing [17:21:22] can you explain? [17:22:42] chasemp: I get the error when I go to http://tools.wmflabs.org/etytree/etymology/resources/html/index.html [17:22:46] and search a word [17:22:49] e.g. door [17:23:30] Epantaleo: works fine for me on OSX with Chrome Version 56.0.2924.87 (64-bit) [17:23:42] Epantaleo: works for me on firefox 51 [17:24:31] the network tab do not seem to contain an OPTIONS so I don't think my request went through the CORS process [17:26:20] oh ok [17:26:27] I reset chrome settings... [17:26:31] now it works [17:26:36] that's weird :) [17:26:42] thanks! [17:27:29] * zhuyifei1999_ wonders what settings affected that [17:28:06] I have the same version of Chrome as you so it souded weird [17:28:12] sounded [18:00:40] 06Labs, 10Labs-Infrastructure, 06Revision-Scoring-As-A-Service, 07artificial-intelligence: GPU resources for Labs - https://phabricator.wikimedia.org/T159165#3058608 (10hashar) Linux Kernel 4.10 added virtual GPU granted you have Intel Haswell with Intel Graphics https://kernelnewbies.org/Linux_4.10#head-6... [18:01:08] 06Labs, 10Labs-Infrastructure, 06Revision-Scoring-As-A-Service, 07artificial-intelligence: GPU resources for Labs - https://phabricator.wikimedia.org/T159165#3065144 (10dr0ptp4kt) I wanted to note something uncovered while looking into this today. TensorFlow CUDA support on Nvidia would currently require... [18:56:08] Hallo. [18:57:17] I'm trying to revive my telegrambot labs project, which I started a few months ago and neglected. It's going surprisingly well, but I have some question. [18:57:22] s [18:58:10] node --version says "v0.10.25"... isnt' that oldish? (Node.js version numbering is a bit confusing to me) [18:58:31] also, is it updated on all instances, or do I have to update my instance myself? [18:58:58] bd808, yuvipanda ^ [19:07:17] aharoni: telegrambot is your project with your own VM? [19:07:31] if so you are on the hook for upgrading and managing [19:08:56] chasemp: yeah, probably [19:09:28] I'm really not that experienced with labs. everything I did there was under support from bd808 at the latest hackathon. [19:09:38] in general nodejs tends to lag behind in official repos, depending on what distor you are running try pulling from backports [19:09:51] !log deployment-prep "mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki he wiktionary hewiktionary he.wiktionary.beta.wmflabs.org" T158628 [19:09:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [19:09:58] T158628: Create beta hewiktionary for testing InterwikiSorting & Cognate - https://phabricator.wikimedia.org/T158628 [19:10:17] aharoni: sure no worries, we are short handed today or I would try to help you figure it out. If you make a ticket on phabricator.wikimedia.org we can try not to forget [19:10:24] but pulling a newer version should be no big deal [19:10:27] depending :) [19:11:02] chasemp: so... do I have to run something like `apt-get update` every now and then? [19:11:49] we do unattended upgrades for security purposes iirc but yeah things get crufty over time [19:11:56] apt-get update just pulls down new package definitions [19:12:10] in general if you service is running and you keep an eye on labs-announce mailing list for changes or relevant things [19:12:14] you should be doing ok [19:15:36] I wouldn't even think of that (I thought that updates are all automatic), but my code uses some npm packages that produce warnings in `npm install`, saying that the node.js version is old. [19:16:32] and if I try to run sudo apt-get, it asks for "password for tools.telegrambot", and I don't even remember setting one. [19:16:42] and if it's not apt-get update, then what should it be? [19:17:53] aharoni: aah, this is a tool [19:18:07] not a labs project (which is what chasemp is talking about) [19:18:18] heh :) [19:18:25] see? I don't really know the difference :/ [19:18:31] project is an overloaded word [19:18:44] aharoni: yeah, it can be confusing! [19:19:14] in that case... is it updated automatically? [19:20:17] aharoni: in taht case aharoni where you are running that to determine current version? [19:20:22] tools-bastion-03? [19:20:48] tools.telegrambot@tools-bastion-03 [19:23:03] aharoni: we have a product that allows tools to be run on debian jessie which should ahve a newer version but I'm unsure if nodejs is fleshed out, madhuvishy do you know or can you ask yuvi? [19:25:03] poking yuvipanda who is next to me [19:25:07] aharoni: details are basically that bastion host is Ubuntu Trusty, but we have possibly the ability for you to run your job on Debian Jessie. Need to verify here [19:25:15] is there a virtualenv thing for nodejs? [19:25:44] I've been poked. [19:27:25] [ reconnecting ] [19:29:05] chasemp: aharoni the nodejs in kubernetes images is newer, but we decided to not build a wrapper script for jobs - we only have it for web services [19:29:18] chasemp: and upgrading node on trusty will break other trusty tools [19:29:28] yeah fair enough [19:29:30] since our current version is ancient and has breaking changes [19:29:37] yuvipanda: there is no virtualenv nodejs thing? [19:29:54] I have little experience in nodejs land honestly [19:30:01] chasemp: I don't know. also virtualenv doesn't give you a new python version - that's what aharoni wants [19:30:13] ...right [19:30:15] ok well! [19:30:17] npm acts like virtualenv by default, installing libraries in current folder under node_modules [19:30:36] yeah, I'm on a big meeting and doing dual mental duty here so my bad of course that's right [19:30:45] me too [19:30:52] silly andrew eating poison food [19:31:01] :D [19:31:05] awww [19:32:03] I think aharoni's options now are learn how to use kubectl, or install nodejs locally [19:32:13] yep [19:32:25] or ask for a project but that seems possibly too broad depending [19:32:30] but anyhoo thanks yuvipanda madhuvishy [19:32:41] * yuvipanda nos [19:32:43] * yuvipanda nods [19:32:48] "learn how to use kubectl" [19:32:52] where do I learn that? [19:33:21] [ [ [ more command line tax? ] ] ] [19:33:53] unfortunately all those options involve more command line tax [19:34:36] https://kubernetes.io/docs/user-guide/kubectl-overview/ but it doesn't talk about the customizations that we've made on tools (specifically you're limited in what images you can run) [19:34:59] aharoni: can you file a bug and I can respond tomorrow? me and madhuvishy are at a conference right now... [19:35:06] well, I'm fine with learning some commands... [19:35:09] and OK, I'll file a bug [19:35:28] aharoni: do read through that link :) [19:35:55] * yuvipanda goes afk again for a bit [19:41:35] 06Labs, 06translatewiki.net: update node.js on tools.telegrambot - https://phabricator.wikimedia.org/T159368#3065513 (10Amire80) [20:15:02] 06Labs, 10Tool-Labs, 06translatewiki.net: update node.js on tools.telegrambot - https://phabricator.wikimedia.org/T159368#3065701 (10scfc) [20:15:19] 06Labs, 10Tool-Labs, 06translatewiki.net: update node.js on tools.telegrambot - https://phabricator.wikimedia.org/T159368#3065513 (10scfc) [20:15:21] 06Labs, 10Tool-Labs: Create Updated NodeJS container for Tool Labs - https://phabricator.wikimedia.org/T155063#3065702 (10scfc) [20:15:52] 06Labs, 10Tool-Labs, 06translatewiki.net: update node.js on tools.telegrambot - https://phabricator.wikimedia.org/T159368#3065513 (10scfc) p:05Triage>03Low [20:17:08] 06Labs: Blacklist apache from unattended-upgrades on tools puppetmaster - https://phabricator.wikimedia.org/T159254#3065710 (10scfc) If `apache2` ist blacklisted, who will inform #Labs/#Tool-Labs administrators of the need to manually update `apache2`? [21:58:42] 10Labs-project-Wikistats, 13Patch-For-Review: miraheze custom domains not updated on wikistats - https://phabricator.wikimedia.org/T158976#3066014 (10Dzahn) 05Open>03Resolved This is finally fixed now. More comments on the change above in Gerrit. "allthetropes" etc are back in the list. [22:01:01] 10Labs-project-Wikistats: miraheze custom domains not updated on wikistats - https://phabricator.wikimedia.org/T158976#3066030 (10Dzahn) [22:43:34] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Baseline77 was created, changed by Baseline77 link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Baseline77 edit summary: Created page with "{{Tools Access Request |Justification=I would like to create a tool that extracts data from the wiktionary (via some scripts) and searches specifically for sports terms via a..." [23:06:15] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Baseline77 was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=1617235 edit summary: [23:27:17] If a tool has been abandoned, but code is available and just needs a new maintainer, which workboard/column should the task-with-details be filed under? (Specifically https://phabricator.wikimedia.org/T152049 which I assume belongs in #tool-labs - but which column would it live under? [23:27:26] ) [23:42:51] 06Labs, 10Tool-Labs, 10Technical-Tool-Request: New Maintainer needed for LanguageTool WikiCheck on Tool Labs - https://phabricator.wikimedia.org/T152049#3066181 (10Quiddity) [23:58:09] quiddity: a question to which I don't know the answer [23:58:27] quiddity: that work board is basically a nightmare and barely used also [23:59:49] chasemp, np, as long as I'm not missing a good solution, I'll take the temp solution of leaving it in that board's backlog. :) (Organization is almost as hard as Documentation!)