[00:25:08] * yuvipanda thanks Matthew_ for taking on https://phabricator.wikimedia.org/T137146 [00:25:39] You're welcome :) [00:40:56] 06Labs: Outgoing mail to wikimedia.org not working on new labs instance - https://phabricator.wikimedia.org/T137160#2359435 (10AlexMonk-WMF) >>! In T137160#2359786, @faidon wrote: > The records that we need are: > ``` > wmflabs.org. 1H IN MX 10 mx1001.wikimedia.org. > wmflabs.org. 1H IN MX 50 mx2001.wikimedia.or... [00:44:38] 06Labs, 10Labs-Infrastructure: wmflabs.org does not resolve - https://phabricator.wikimedia.org/T38885#418242 (10AlexMonk-WMF) I'm guessing we lost the status change history in the migration from BZ, but I don't think this is resolved. We still don't have A/AAAA records for wmflabs.org. itself, we should be ab... [00:59:07] 06Labs, 10wikitech.wikimedia.org, 15User-bd808: Create Scribunto extension that allows lua to get LDAP data for wikitech - https://phabricator.wikimedia.org/T127626#2359891 (10bd808) a:05bd808>03None [00:59:19] 06Labs, 10wikitech.wikimedia.org: Create Scribunto extension that allows lua to get LDAP data for wikitech - https://phabricator.wikimedia.org/T127626#2048983 (10bd808) [01:37:33] 10Quarry: Setup an easy way to have Quarry dump information / results on a wiki page - https://phabricator.wikimedia.org/T137179#2359905 (10yuvipanda) [01:38:19] 10Quarry: Setup an easy way to have Quarry dump information / results on a wiki page - https://phabricator.wikimedia.org/T137179#2359918 (10yuvipanda) [01:43:30] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Music1201 was created, changed by Music1201 link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Music1201 edit summary: Created page with "{{Tools Access Request |Justification=To create a bot planned to be ran on enwiki using pywiki |Completed=false |User Name=Music1201 }}" [02:10:41] 06Labs: Outgoing mail to wikimedia.org not working on new labs instance - https://phabricator.wikimedia.org/T137160#2359920 (10dpatrick) 05Open>03Resolved This is working now. Thank you! [02:20:25] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Music1201 was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=615882 edit summary: [08:44:11] 10Tool-Labs-tools-Other, 06Commons: Flickr2Commons throttle needed - https://phabricator.wikimedia.org/T100973#2360186 (10Pokefan95) [11:54:47] Is it possible to increase disk space on a labs machine? Trying to balance the cost of optimizing code vs just adding more hardware, for Phlogiston. [11:58:19] jaufrecht: if you rebuild the instance, yes [14:29:30] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Mattias Östmar (WMSE) was created, changed by Mattias Östmar (WMSE) link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Mattias_%c3%96stmar_(WMSE) edit summary: Created page with "{{Tools Access Request |Justification=Get access to the WLM database in relation to the Connected Open Heritage Project. |Completed=false |User Name=Mattias Östmar (WMSE) }}" [15:01:18] 10Tool-Labs-tools-Other: TextCatDemo 500/503 errors - https://phabricator.wikimedia.org/T137085#2361071 (10TJones) I've reviewed the demo on IE10 on Windows 8; Edge12 on Windows 10; Chrome, FF, and Opera on Ubuntu; and Safari on an iPhone. I found problems with the demos on the iPhone (Safari zooms in when you... [15:03:09] jaufrecht: are you using the instance extended disk space alreaady? [15:03:17] jaufrecht: you can get a second partition mounted on /srv [15:03:40] role::labs::lvm::srv [15:04:08] currently: /dev/vda1 18569668 8902256 8701080 51% / [15:04:10] you would get /dev/vda1 19G on / [15:04:27] and /dev/mapper/vd-second--local--disk Lot of G /srv [15:04:41] the default for / is 20Gbytes iirc [15:04:47] so primary is udev and secondary is /dev/vda1 ? [15:04:54] but different flavor have different disk space, it remains unallocated by default [15:04:56] I already have /dev/vda1, but I'm not using /dev [15:05:07] yeah /dev/vda1 is the default [15:05:15] sure, I could use a second one [15:05:17] if you have nothing in /srv you can try [15:05:49] in wikitech for your labs project you would have to add the puppet class role::labs::lvm::srv somewhere at https://wikitech.wikimedia.org/wiki/Special:NovaPuppetGroup [15:05:54] I have nothing in /srv. what's the command? [15:05:58] then the class would be available to be applied on instances [15:06:12] then run puppet and it will take care of mounting the extra disk on /srv [15:06:21] (running puppet being: sudo puppet agent -tv [15:18:19] /dev/mapper/vd-second--local--disk only really works if your instance is larger than m1.small [15:18:49] m1.small's /dev/mapper/vd-second--local--disk is negligible [15:25:10] jaufrecht: if you're referring to https://wikitech.wikimedia.org/wiki/Nova_Resource:Phlogiston, that works best on https://wikitech.wikimedia.org/wiki/Nova_Resource:Phlogiston-2.phlogiston.eqiad.wmflabs [15:25:36] Cool. Confused by the relationship between "the puppet class role::labs::lvm::srv", NovaPuppetGroup, and the "phlogiston" project. where does Phlogiston fit in? [15:26:47] what do you mean? role::labs::lvm::srv is a puppet role, which after a while (or immediately if you run sudo puppet agent -tv) it will apply some configs to the server [15:27:37] (in that case it creates and mounts the extra disk space (about 140G for Phlogiston-2)) [15:28:04] NovaPuppetGroup is a special page to configure puppet groups for instances [15:28:31] phlogiston is a project I guess... [15:29:51] Is "role::labs::lvm::srv" a role you just set up, which is specific to Phlogiston-2? [15:31:08] you're project admin, you can go to https://wikitech.wikimedia.org/wiki/Special:NovaPuppetGroup to setup [15:31:30] wait not there [15:31:34] that is, what am I applying that role _to_? The only things I can do on that page ... [15:33:26] jaufrecht: go to https://wikitech.wikimedia.org/wiki/Special:NovaInstance -> set "phlogiston" in the project filter -> find the instance you want to change the in the table below that -> click "configure" to get to the Puppet management screen for that instance [15:33:52] yeah that [15:34:12] then check the box for "role::labs::lvm::srv" and click "submit" [15:34:27] found it [15:34:32] then ssh to the instance and run `sudo puppet agent --test --verbose` [15:34:55] then write this up on wikitech :) [15:35:12] or use !log lol [15:35:53] success [15:35:56] will write it up [15:36:25] Labs or Tool Labs? [15:36:32] bd808: has the instance suspending and resuming started? I see an instance of mine mysteriously in "SHUTOFF" state [15:36:36] jaufrecht: labs [15:37:14] zhuyifei1999_: I'm not sure. andrewbogott is the one who would know [15:37:20] whenever you deal with instances / puppets, it's usually labs, unless you're tool labs admin [15:37:56] zhuyifei1999_: probably unrelated. What instance? [15:38:09] https://wikitech.wikimedia.org/wiki/Nova_Resource:Encoding03.video.eqiad.wmflabs [15:38:19] per https://wikitech.wikimedia.org/wiki/Special:NovaInstance [15:38:29] (haven't checked ssh yet_ [15:38:31] *) [15:39:30] I don't know why it's shut down, but I started it. [15:39:35] ok [15:39:39] thx [15:40:10] 06Labs, 10Labs-Infrastructure, 10Monitoring, 07Blocked-on-Operations: Provide a grafana installation for labs - https://phabricator.wikimedia.org/T137216#2361154 (10akosiaris) [15:40:52] 06Labs, 10Labs-Infrastructure, 10Monitoring, 07Blocked-on-Operations: Provide a grafana installation for labs - https://phabricator.wikimedia.org/T137216#2361167 (10akosiaris) p:05Triage>03Normal [15:44:53] documented: https://wikitech.wikimedia.org/wiki/Help:Adding_Disk_Space. thanks. [15:45:38] jaufrecht: thanks for the writeup [15:48:55] * zhuyifei1999_ thinks "Mounting additional disk space" or something similar would describe it more accurately [15:49:26] I'm going to add some prose and find some places to link it to [15:51:03] wow https://wikitech.wikimedia.org/wiki/Help:Resize_root_partition [15:51:42] bd808: perhaps this too (found it via special:search on "role::labs::lvm::srv") [15:51:54] *nod* [15:52:50] I linked it from https://wikitech.wikimedia.org/wiki/Help:Contents [16:02:15] andrewbogott, deployment-cache-upload04 is restarting? [16:02:48] Krenair: could you tell me more about what you're seeing? [16:03:02] I got shinken alert emails saying it was down [16:03:14] It was probably suspended briefly [16:03:14] looking at the log in horizon, apparently it was suspended and then resumed by novaadmin [16:03:19] but then it stopped [16:03:22] unless you see other evidence that it's broken/unhappy [16:03:25] with a user ID of '-' [16:03:52] yeah http://upload.beta.wmflabs.org/ is broken [16:03:56] no ping [16:04:14] https://lists.wikimedia.org/pipermail/labs-announce/2016-June/000125.html [16:04:22] Is anything happening that isn't explained by ^ ? [16:04:38] I saw that [16:04:40] but then it stopped [16:05:30] Request ID [16:05:30] Action [16:05:30] Start Time [16:05:31] User ID [16:05:31] Message [16:05:35] req-bfabd4e0-181b-4fd9-ae0b-cce80f36ec68 Stop June 7, 2016, 3:59 p.m. - - [16:05:37] ah, ok [16:05:55] Apparently labvirt1010 is very upset and stopping instances instead of suspending them [16:06:04] not sure what that's about, I'll stop my script and investigate [16:06:05] the two lines below that are the suspend and resume [16:47:28] 06Labs: Replicate watchlist to labs - https://phabricator.wikimedia.org/T93887#2361355 (10jcrespo) [16:47:33] 06Labs, 10labs-sprint-116, 10DBA, 13Patch-For-Review: Make watchlist table available on labs - https://phabricator.wikimedia.org/T59617#2361357 (10jcrespo) [16:56:11] 06Labs, 10labs-sprint-116, 10DBA, 13Patch-For-Review: Make watchlist table available on labs - https://phabricator.wikimedia.org/T59617#2361443 (10jcrespo) After some thinking, and my thoughts on why labs breaks so easily T136618#2356834, I think this is one of the cases that would fail to be replicated ac... [17:13:09] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Security-Reviews: Security review of Tool Labs console application - https://phabricator.wikimedia.org/T135784#2361564 (10bd808) @dpatrick and @Bawolff Can I get a guesstimate on when you might get to this review? The core code is complete for the initial... [17:44:51] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Mattias Östmar (WMSE) was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=618713 edit summary: [18:00:13] 10Tool-Labs-tools-Other, 06Community-Tech-Tool-Labs, 15User-bd808: Create temporary http -> https reverse proxy for MerlBot - https://phabricator.wikimedia.org/T137235#2361722 (10bd808) [18:02:34] 10Tool-Labs-tools-Other, 06Community-Tech-Tool-Labs, 15User-bd808: Create temporary http -> https reverse proxy for MerlBot - https://phabricator.wikimedia.org/T137235#2361737 (10bd808) Longer term fix is updating the bot to use the JDK8 reliant library mentioned in {T121279} [18:09:32] 10Tool-Labs-tools-Other, 06Community-Tech-Tool-Labs, 15User-bd808: Create temporary http -> https reverse proxy for MerlBot - https://phabricator.wikimedia.org/T137235#2361722 (10bd808) p:05Triage>03High [19:36:51] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Nodepool can barely spawn instances OpenStack - https://phabricator.wikimedia.org/T137241#2361986 (10hashar) [19:37:29] andrewbogott: somehow Nodepool has troubles spawning instances :( Though sometime it works.. https://phabricator.wikimedia.org/T137241 [19:37:43] hashar: yeah, it's a known problem, I'm working on it [19:37:46] known but not understood :( [19:38:13] andrewbogott: good to know it is known :D At least the nova api calls seems to work fine :) [19:41:10] 06Labs, 10Labs-Infrastructure, 10Monitoring, 07Blocked-on-Operations: Provide a grafana installation for labs - https://phabricator.wikimedia.org/T137216#2361154 (10hashar) > Remove labmon1001 as a data source from the production grafana installation. I would keep it around if at all possible. For people... [19:43:58] hashar: somehow rabbitmq for labvirt1010 is gummed up. It connects just fine, but no messages pass back and forth [19:46:31] andrewbogott: I would blame puppet / package upgrading and in last resort restart of services. Then I have no experience with openstack :( [19:58:20] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Security-Reviews: Security review of Tool Labs console application - https://phabricator.wikimedia.org/T135784#2362070 (10dpatrick) @bd808 How does the week of July 5th and into part of the week of the 11th work for you? That's the earliest we can schedule... [20:02:16] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Nodepool can barely spawn instances OpenStack - https://phabricator.wikimedia.org/T137241#2362081 (10hashar) @Andrew knows about the issue > somehow rabbitmq for labvirt1010 is gummed up. It connects just fine, but no messages pass bac... [20:11:03] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Security-Reviews: Security review of Tool Labs console application - https://phabricator.wikimedia.org/T135784#2362138 (10bd808) >>! In T135784#2362070, @dpatrick wrote: > @bd808 How does the week of July 4th and into part of the week of the 11th work for... [20:12:01] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Diffusion: Deploy "Striker" Tool Labs console to WMF production - https://phabricator.wikimedia.org/T136256#2362142 (10bd808) Security review is tentatively scheduled for July 5-15. [20:29:30] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Diffusion: Deploy "Striker" Tool Labs console to WMF production - https://phabricator.wikimedia.org/T136256#2362190 (10bd808) >>! In T136256#2329954, @Krenair wrote: >> Future services: >> Nova api > > Does it need to be in the labs support network then?... [20:47:09] 06Labs, 06Discovery, 06Operations, 10hardware-requests: rack/upgrade/setup/install/deploy relforge100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T136708#2362279 (10RobH) [21:06:32] andrewbogott: I'm trying to test a fix by creating a test branch on labstestweb2001 under /srv/mediawiki but something is deleting it every few minutes (puppet? rsync?) any idea how to disable that? [21:07:57] tgr: I don't know. puppet is currently disabled on that box, so it's not puppet [21:14:24] 10PAWS: download datafiles generated in PAWS - https://phabricator.wikimedia.org/T137251#2362335 (10Capt_Swing) [21:21:45] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Diffusion: Deploy "Striker" Tool Labs console to WMF production - https://phabricator.wikimedia.org/T136256#2362353 (10yuvipanda) That comment of mine was way less clearer than I intended. What I meant was that the tools specific functionality should be li... [21:44:37] 06Labs, 06Discovery, 06Operations, 10hardware-requests: rack/upgrade/setup/install/deploy relforge100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T136708#2362400 (10RobH) [21:46:38] 06Labs, 06Discovery, 06Operations, 10hardware-requests, 03Discovery-Search-Sprint: eqiad: (2) Relevance forge servers - https://phabricator.wikimedia.org/T131184#2362408 (10RobH) 05stalled>03Resolved a:03RobH The two machines have been allocated and are now in the OS installation stage. I'm resolv... [21:47:01] 10Quarry: Add date when query was last run - https://phabricator.wikimedia.org/T77941#832144 (10Capt_Swing) Hey @yuvipanda, possible we could get this implemented in the near future? Every time I use Quarry and share the results with anyone, I wish for this feature. [21:59:41] 06Labs, 06Discovery, 06Operations, 10hardware-requests: rack/upgrade/setup/install/deploy relforge100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T136708#2362475 (10RobH) [22:03:13] 06Labs, 06Discovery, 06Operations, 10hardware-requests: rack/upgrade/setup/install/deploy relforge100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T136708#2362483 (10RobH) [22:03:28] 06Labs, 06Discovery, 06Operations: rack/upgrade/setup/install/deploy relforge100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T136708#2345045 (10RobH) [22:05:57] 06Labs, 06Discovery, 06Operations: rack/upgrade/setup/install/deploy relforge100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T136708#2362489 (10RobH) [22:06:41] 06Labs, 06Discovery, 06Operations: rack/upgrade/setup/install/deploy relforge100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T136708#2345045 (10RobH) a:05Cmjohnson>03EBernhardson All tasks for the setup of relforge100[12] have been accomplished, other than service implementation. I'm assigning t... [22:16:22] 10Quarry: Add date when query was last run - https://phabricator.wikimedia.org/T77941#2362536 (10yuvipanda) I'll try to get to it before end of next week. [22:22:05] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Nodepool can barely spawn instances OpenStack - https://phabricator.wikimedia.org/T137241#2362566 (10Andrew) 05Open>03Resolved a:03Andrew This turned out to not be a rabbitmq issue; rather, libvirtd was having some kind of crisis b... [22:23:11] 06Labs, 06Discovery, 06Operations, 10hardware-requests, 03Discovery-Search-Sprint: eqiad: (2) Relevance forge servers - https://phabricator.wikimedia.org/T131184#2362575 (10EBernhardson) [22:23:13] 06Labs, 06Discovery, 06Operations: rack/upgrade/setup/install/deploy relforge100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T136708#2362571 (10EBernhardson) 05Open>03Resolved Thanks! All the work here is much appreciated. I've put together T137256 to track setting up the elasticsearch cluster an... [23:47:48] yuvipanda: i am using horizon the first time to add a webproxy to an instance. i see the form where i can add a proxy, but what needs to happen for my instance to showup in the list of instances to pick from as backends [23:48:59] yuvipanda: ignore, i get it now [23:50:54] added proxy.. though it does not appear to exist yet