[00:11:06] were r you [00:24:09] yea were is home [00:25:39] which one noo or yea [00:36:06] ok anonymous I assume ...introducing my first day. so hello everyone [00:42:47] by chance you talking bout me ( spawn) [00:45:41] (/embed ) [00:51:42] 10Wikimedia-Labs-wikitech-interface, 10Wikimedia-Site-requests: Investigate the creation of a wiki for the dev community that is organizing around Labs - https://phabricator.wikimedia.org/T70818#1219238 (10Krenair) [01:05:52] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/I.yeckehzaare was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=154525 edit summary: [02:30:02] Krenair: I don't get it [02:30:59] You definitely don't have bast1001 access [02:31:11] haha :( [02:31:34] *definitely* [02:34:14] Krenair: its not like there actually is anything special on bast1001 just the access that it gives (and not even that) [10:21:38] (03PS1) 10Werdna: Add Blueprint skin to notify list for #wikimedia-design [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/205221 [10:22:29] (03PS1) 10Werdna: Add #wikimedia-design configuration [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/205222 [10:45:07] 6Labs, 7Tool-Labs, 10Wikimedia-Hackathon-2015: Organize Wikimedia Labs activities at the Wikimedia Hackathon 2015 - https://phabricator.wikimedia.org/T92274#1220356 (10Qgil) In relation to {T76325}, are there any specific developers of any related communities that you would like to invite to Lyon? We can tal... [11:14:28] (03CR) 10Merlijn van Deen: [C: 032] Add Blueprint skin to notify list for #wikimedia-design [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/205221 (owner: 10Werdna) [11:14:41] (03Merged) 10jenkins-bot: Add Blueprint skin to notify list for #wikimedia-design [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/205221 (owner: 10Werdna) [11:17:33] (03CR) 10Nemo bis: [C: 04-1] "I think removing stuff from the main channel and hiding it on some side channel is a regression, as long as the channel in question is not" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/205222 (owner: 10Werdna) [11:18:28] !log tools.wikibugs Updated channels.yaml to: 8e88fc89deaa41b2a720845f5d20aa871ffa09d9 Add Blueprint skin to notify list for #wikimedia-design [11:18:35] Logged the message, Master [11:42:33] (03CR) 10Werdna: "If something is being removed from the main channel, it's unintentional." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/205222 (owner: 10Werdna) [11:42:56] Nemo_bis: “Removing from the main channel”? [11:43:53] werdna: unless things changed, adding a specific channel for a project removes that project from #wikimedia-dev [11:44:04] (or you have to specify the project for both channels) [11:44:05] Nemo_bis: how can I change that? [11:44:16] See last message [11:44:28] I've not checked in few months though, things might have changed [11:44:59] Nemo_bis: does this help? [11:44:59] (03PS2) 10Werdna: Add #wikimedia-design configuration [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/205222 [12:58:43] can someone restart catscan? [12:59:14] comets: Which one? [12:59:44] http://tools.wmflabs.org/catscan2/catscan2.php? [13:00:26] That's odd - the webserver is actually running but not proxied. [13:00:29] * Coren kicks it. [13:09:08] It lives. [13:16:18] 7Tool-Labs: tools-redis broken - https://phabricator.wikimedia.org/T96485#1220481 (10coren) ```[3275840.851752] Out of memory: Kill process 27882 (redis-server) score 932 or sacrifice child``` Unambiguously tells us that the OOM killer was woken up - so there is a point at which memory use increases to the poin... [13:24:35] u killed it :( [13:25:47] *Ķābööm* [13:25:50] nvm :) [13:25:55] lol [13:41:40] 7Tool-Labs: Multiple queue runners on tools-mail - https://phabricator.wikimedia.org/T74867#1220505 (10scfc) @valhallasw: T96548 is a duplicate; please unblock that and merge it in here. [13:53:37] 7Tool-Labs: Multiple queue runners on tools-mail - https://phabricator.wikimedia.org/T74867#1220537 (10valhallasw) [13:54:50] 7Tool-Labs: Multiple queue runners on tools-mail - https://phabricator.wikimedia.org/T74867#770309 (10valhallasw) {T96548} is a specific example for this; it's restricted because it contains potentially private information (email addresses and times at which emails were sent) [14:17:33] gifti: Are you ready to reboot your instance? [14:30:10] 6Labs, 6operations, 7network: Slow network between virt nodes - https://phabricator.wikimedia.org/T96561#1220645 (10Andrew) 3NEW [14:33:09] 6Labs, 6operations: Can't create new trusty or jessie instances on labvirt boxes - https://phabricator.wikimedia.org/T96563#1220665 (10Andrew) 3NEW [14:34:13] 6Labs, 6operations: Can't create new trusty or jessie instances on labvirt boxes - https://phabricator.wikimedia.org/T96563#1220677 (10Andrew) Things I've checked already: - I can still create working instances on virt100x boxes. - Firewall settings don't seem to distinguish between virt100x and labvirt100x [14:38:51] 6Labs, 6operations: Can't create new trusty or jessie instances on labvirt boxes - https://phabricator.wikimedia.org/T96563#1220695 (10Andrew) My instinct is to suspect https://gerrit.wikimedia.org/r/#/c/205093/ but so far I can't provide any evidence that that's the culprit. [14:59:00] 6Labs, 6operations: Can't create new trusty or jessie instances on labvirt boxes - https://phabricator.wikimedia.org/T96563#1220737 (10Andrew) Andrews-MBP-2:~ andrew$ ssh virt1000.wikimedia.org root@virt1000:~# source ~/novaenv.sh root@virt1000:~# nova boot --image e9587207-1fa2-4faf-926b-d9efc41b176c --flavo... [15:08:27] 7Tool-Labs, 5Patch-For-Review: webservice creates blocking files and jobs when called from a user account with an eponymous tool - https://phabricator.wikimedia.org/T66219#1220744 (10scfc) 5Open>3Resolved [15:14:04] 7Tool-Labs, 5Patch-For-Review: Epilog scripts for web services fail with exit codes 1 and 255 - https://phabricator.wikimedia.org/T96491#1220747 (10scfc) 5Open>3Resolved The case of normal users trying to start web services is now resolved; I note that the only issues with tool accounts were on April 15th.... [15:24:59] !log grantreview Updated /vagrant to latest HEAD (e9856d6) and provisioned [15:25:05] Logged the message, Master [15:31:05] !log grantreview Application code is now in /vagrant/srv/iegreview due to mediawiki-vagrant layout changes [15:31:08] Logged the message, Master [15:35:13] !log grantreview Updated application to 7303e5a (Update wikitext report for Inspire) [15:35:15] Logged the message, Master [15:53:51] hashar: Are there any precise instances left in deployment-prep noawadays? [15:55:01] Coren: most probably [15:55:05] integration has a bunch as well [15:56:12] for integration we have 4 Precise instances being used as Jenkins slaves [15:56:16] mostly to run PHP 5.3 tests [15:56:21] * Coren nods. [15:56:49] and the puppetmaster of both projects are still Precise [15:56:56] cant run it with Trusty [15:57:15] We need to schedule a reboot for them if they haven't been rebooted since https://gerrit.wikimedia.org/r/#/c/203384/ has been merged. [15:57:39] on deployment-prep there are the varnish caches, memcached, databases, the bastion and a few mores [15:58:09] what is that nfs idmap thing ? :) [15:58:56] integration barely rely on nfs [15:58:56] but some deployment-prep instances uses /data/project [15:58:57] nfs 4 has two ways to pass around auth; by username or user id. Precise instances used the former, we're switching them to the latter. [15:59:09] So that all releases use ids. [16:00:05] It's mostly a noop, but needs to be done so that we can turn off the matching feature on the server. [16:00:13] ohh [16:00:42] we had an issue with inconsitent UID on beta [16:02:01] Coren: I think the apache / mwdeploy users and so on were having different uid [16:02:10] so I suspect idmap is used to map them to the same uid [16:02:20] or maybe the users have been defined on the nfs server [16:02:50] It's probably a bit of both - part of the issue is having to uniformize all of this. [16:03:24] NFS should only be used for users made consistent either through puppet (like prod) or ldap. [16:05:01] I guess we will have to reboot instances and see what happens [16:05:21] I am rebasing the beta cluster puppetmaster [16:06:37] I sent a rather detailed email about the uid issue to the labs lists last week. :-) [16:10:46] Coren: yeah too many mails :) [16:10:52] will give a shot tomorrow. I am off for now [16:11:01] feel free to poke folks in #wikimedia-releng about it though [16:16:19] Krinkle: quick question: do we ever match hosts with this first pattern? https://github.com/Krinkle/dotfiles/blob/master/modules/functions.sh#L27 [16:20:44] 6Labs, 6operations: Can't create new trusty or jessie instances on labvirt boxes - https://phabricator.wikimedia.org/T96563#1220936 (10Andrew) On labnet1001, we get dhcp requests for precise on labvirt, trusty on virt, but not for trusty on labvirt. Here's trusty on virt: Apr 20 16:13:08 labnet1001 dnsmasq-d... [16:21:17] (03CR) 10Awight: "ping... This will require manual rebasing until merged or abandoned..." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/196852 (owner: 10Awight) [16:57:28] 6Labs, 5Patch-For-Review: labvirt boxes need a new cert for libvirtd - https://phabricator.wikimedia.org/T96291#1221067 (10akosiaris) 5Open>3Resolved Certs successfully issued from privated CA and populated on servers, resolving [17:11:39] 6Labs, 6operations: Can't create new trusty or jessie instances on labvirt boxes - https://phabricator.wikimedia.org/T96563#1221112 (10Andrew) When I try to mount the drive of a failed instance on labvirt, I get: root@labvirt1001:/var/lib/nova/instances/7fc33e19-b002-4965-b7a4-886ea824fd9b# mount /dev/nbd0 /t... [17:20:42] 6Labs, 3Labs-Q4-Sprint-2, 3Labs-Q4-Sprint-3, 3ToolLabs-Goals-Q4: Do a rolling restart of Tool Labs precise instances - https://phabricator.wikimedia.org/T95557#1221156 (10coren) Poking maintainers for a last chance to reboot on their own schedule. [17:21:24] Coren: xtools or wikiviewstats on that list? [17:21:50] T13|detached: I'd need to check, gimme a sec. I think xtools is okay [17:22:46] T13|detached: Ah, no - I remembered wrong. xtools has 5 instances needing a reboot. [17:23:04] wikiviewstats is okay. [17:23:11] 5? [17:23:24] Hrmm. I only can think of 4. [17:23:27] T13|detached: exec-light webservice2 exec-heavy webservice and login [17:24:04] I'll go do the webservice restarts [17:24:12] Ima send email to the labs lists shortly with the list. [17:29:35] Coren: {{Done}} [17:34:17] 6Labs, 7Tool-Labs, 3Labs-Q4-Sprint-1, 3Labs-Q4-Sprint-2, 3ToolLabs-Goals-Q4: Make sure tools-db is replicated somewhere - https://phabricator.wikimedia.org/T88718#1221188 (10coren) Tools-db is now properly puppetized; I'ma create a changeset for review by Sean to create the matching slave this week. [17:34:38] Hi [17:35:10] I need a guide to enable php and make a query over wiki, do you have one? [17:37:20] or over phyton... or anything else XD [17:37:35] Hprmedina: have you looked at https://wikitech.wikimedia.org/wiki/Help:Tool_Labs ? [17:37:50] I'm there! [17:37:59] what help do you need? [17:38:17] a shortcut? XD [17:38:46] i'm flyin over several links.. and I don't arrive :P [17:39:14] I have already the tool [17:39:26] http://tools.wmflabs.org/grillitus/ <- that is working [17:39:37] Hprmedina: are you logged in via ssh? [17:39:43] yes [17:40:09] if you just need to run some queries, you can use the mariadb client [17:40:39] yes, I did that, but I want to put some querys over http [17:40:52] or webservice [17:41:36] that sounds a page with php with some code to query the database and put the data over the web [17:41:42] Hprmedina: php is enabled by default [17:42:05] o.o [17:43:13] omg.... my fault!!! [17:43:19] ahahaha.... [17:43:22] :) [17:43:45] I wrote <%php echo 'hi' %> ahahaha.... [17:44:01] now working :P [17:44:30] great [17:45:01] a lot of time without not using php... [17:46:26] Coren: is James on IRC (so I can poke him to close that abandoned labs tools RfC)? [17:46:45] Which James? [17:47:14] can someone remind me again what page on wikitech documents updating puppetmasters for beta? [17:47:36] lol I don't remember which one you said but it wasn't James_F. [17:48:46] Ah, James Alexander. No, I don't see him online atm. [17:48:53] But it's early Pacific time [17:49:31] 10:49 is early? lol okay. [17:49:55] ah found it I think: https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/How_code_is_updated#Updating_puppet_manifests_to_match_production [17:50:42] Ah, hm. Yeah, forgot it's not early anymore. :-) [17:50:54] I still don't see him online. [17:52:47] lol [18:09:12] hello there [18:09:52] I have created a user database but when I try to make a query on MySQL Workbench, I am getting "access denied" error [18:11:38] what to do? [18:31:15] 6Labs: Zillion expired tokens in keystone database - https://phabricator.wikimedia.org/T96256#1221535 (10Andrew) Now 3332254 [18:34:50] hello? [18:37:45] anyone there? [18:38:42] * T13|detached doesn't see anyone. [19:04:33] 6Labs, 6operations, 7network: Slow network between virt nodes - https://phabricator.wikimedia.org/T96561#1221815 (10Andrew) 5Open>3Invalid a:3Andrew Two things were happening here -- half the problem was I forgot the difference between Mb and MB, which makes the picture much better than I thought. 56M... [19:05:15] Superyetkin: Are you certain you are using the right username/password as there is in your tool's .replica.my.cnf? [19:09:42] 10MediaWiki-extensions-OpenStackManager, 7Easy, 7I18n, 5Patch-For-Review: OpenStackManager: Missing message "<novaresources>" on Special:SpecialPages - https://phabricator.wikimedia.org/T69852#1221867 (10ImPacific) 5Open>3Resolved [19:10:31] yes, I am using the username/password in my tool's replica.my.cnf [19:27:26] 6Labs, 6operations: Can't create new trusty or jessie instances on labvirt boxes - https://phabricator.wikimedia.org/T96563#1222008 (10Andrew) 5Open>3Resolved a:3Andrew This was a result of corrupt files in the base-image cache in /var/lib/nova/instances/_base. I've cleaned out the base images for trust... [19:51:46] YuviPanda: what is the best way to deploy a patch that isn't submitted for review such as Matma's [19:59:00] YuviPanda: dplbot webserver continues going unresponsive (appears to be up, but HTML queries return "No webservice" error page) - had to restart it manually 6-8 times over the weekend and today [20:13:35] andrewbogott: good morning :) [20:14:45] ‘morning! [20:18:30] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 5Continuous-Integration-Isolation: OpenStack API account to control `contintcloud` labs project - https://phabricator.wikimedia.org/T86170#1222249 (10hashar) From labnodepool I tried to authenticate using the user/pass I have created via w... [20:19:12] hashar: sorry, I will try to get to that later today [20:20:47] andrewbogott: yeah I have just found out I have "nova" installed on labnodepool [20:20:52] quite handy to test! [20:26:57] Negative24: self hosted puppetmaster, I presume phab-01 has on? [20:28:26] YuviPanda: I don't think so. I doesn't have puppetmaster self configured [20:28:32] *it [20:28:42] there should be docs on wikitech [20:28:58] I have phab-pup with it on and with phab configured though [20:29:17] if matma wants to use it [20:38:11] YuviPanda: Down to 108 to reboot from 225 now [20:38:33] \o/ [20:38:33] cooll [20:42:37] andrewbogott: how’s designate doing? stable enough for us to plan to switch tools over next week? [20:42:51] the old hostname format will still work, right? [20:42:55] even if we switch? [20:43:05] * YuviPanda wants to switch so we can get labsdb and what not out of /etc/hosts [20:43:30] * YuviPanda has to restart dnsmasq now sigh [20:43:39] YuviPanda: I think it would be ok to switch. You could make a test exec node and verify... [20:44:00] I’ve a test node already that I can use, probably (tools-bastion-02, heir to tools-dev) [20:45:16] So, yeah, it seems stable. Of course we haven’t seen it run under load, but… we just don’t produce that much load. [21:04:46] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 5Continuous-Integration-Isolation: OpenStack API account to control `contintcloud` labs project - https://phabricator.wikimedia.org/T86170#1222365 (10Andrew) Here's a test... can you log in to https://horizon.wikimedia.org/ as yourself? (I... [21:18:13] 10Wikimedia-Labs-wikitech-interface, 10Wikimedia-Site-requests: Investigate the creation of a wiki for the dev community that is organizing around Labs - https://phabricator.wikimedia.org/T70818#1222401 (10Negative24) And with OpenStack's Horizon on the horizon (:P) Wikitech should become less crowded with tec... [22:13:27] Coren: I think the precise webhosts are overwhelmed. Some heavy users seem to keep getting half-killed in some form... [22:13:30] * YuviPanda moves them to trusty [22:38:35] 7Tool-Labs: Audit redis usage on toollabs - https://phabricator.wikimedia.org/T91979#1222737 (10yuvipanda) a:5coren>3valhallasw [22:55:48] 7Tool-Labs: Webservices get unregistered with proxy randomly - https://phabricator.wikimedia.org/T96625#1222754 (10yuvipanda) 3NEW [22:58:11] 7Tool-Labs, 5Patch-For-Review: Webservices get unregistered with proxy randomly - https://phabricator.wikimedia.org/T96625#1222769 (10yuvipanda) ^ will restart them as well, which is a fine temporary solution. [23:26:04] YuviPanda, or val[tab]damnit! - wikibugs is AWOL [23:29:49] :P not even `cloud [23:31:09] quiddity: yes, redis disruption. give it a moment. [23:31:13] also it should be free of redis!!! [23:31:20] legoktm: ^ [23:31:25] hi [23:31:28] anyway, I’m cleaning out redis spam from notconfusing [23:31:34] YuviPanda: why should it be free? [23:31:35] we’ll get 11G of space on it back [23:31:42] hh [23:31:44] heh [23:31:52] legoktm: nobody else except wikibugs is using the redis pubsub feed [23:31:58] legoktm: why not just passi t to you as a funciton call [23:32:22] because I don't want to touch it [23:35:46] haha [23:35:47] :D [23:37:24] legoktm: can you bring it back up? redis should be more stable now [23:38:14] see http://graphite.wmflabs.org/render/?width=586&height=308&_salt=1429573085.202&target=tools.tools-redis.redis.6379.db0.keys [23:39:06] sitic: ^ (redis should be much better now)