[00:00:45] I want devhub.wmflabs.org like http://livingstyleguide.wmflabs.org , for Data and developer hub prototype [00:01:05] spagewmf_: ah, so I suggest either using toollabs or just requesting a project :) sidebar of wikitech [00:06:37] yuvipanda: is there a way to see all projects in labs? they're in https://wikitech.wikimedia.org/wiki/Category:Nova_Resource with 500 other things. [00:06:37] spagewmf_: https://wikitech.wikimedia.org/w/api.php?action=query&list=novaprojects [00:06:38] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:06:38] ouashfolushgolsjfhgasfgsagsagf [00:06:38] now what [00:06:38] intermittent hiccup [00:07:28] heh, I was trying to craft the inverse hasProperty:Project Semantic MediaWiki query using my 2007-era knowledge :) [00:11:05] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 767768 bytes in 7.714 second response time [00:29:21] 10Tool-Labs, 5Patch-For-Review: Error mails from SGE are encoded as application/octet-stream - https://phabricator.wikimedia.org/T63160#1263414 (10scfc) 5Open>3Resolved [00:41:22] 6Labs, 10dev.wikimedia.org, 7Tracking: Create Labs project for "Data and developer hub" prototype - https://phabricator.wikimedia.org/T98275#1263446 (10Spage) 3NEW [00:54:02] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [00:55:35] spagewmf_: :D [00:56:15] 6Labs, 10dev.wikimedia.org, 7Tracking: Create Labs project for "Data and developer hub" prototype - https://phabricator.wikimedia.org/T98275#1263558 (10yuvipanda) @spage can you mark it more specific than just 'documentation'? That was the problem with the previous project, I guess :D [00:56:53] yuvipanda: I added your api query to https://wikitech.wikimedia.org/wiki/Help:Getting_Started#Join_Projects , FYI "Contact the current project members" in that section is an SMW query for project members. [00:57:07] spagewmf_: ah cool. [00:57:11] spagewmf_: SMW sucks [00:58:05] yup, SMW is cool and it sucks. We'll see if Wikidata Query Language can do better :) [00:59:38] spagewmf_: :) [00:59:46] spagewmf_: did you accept the invite for tomorrow? [01:01:01] 6Labs, 10Labs-Infrastructure, 3ToolLabs-Goals-Q4: Limit NFS bandwith per-instance - https://phabricator.wikimedia.org/T98048#1263608 (10yuvipanda) [01:20:59] FYI I made https://wikitech.wikimedia.org/wiki/New_Project_Request nicer [01:21:45] spagewmf_: <3 [01:21:56] 6Labs, 10dev.wikimedia.org, 7Tracking: Create Labs project for "Data and developer hub" prototype - https://phabricator.wikimedia.org/T98275#1263673 (10Spage) >>! In T98275#1263558, @yuvipanda wrote: > @spage can you mark it more specific than just 'documentation'? That was the problem with the previous proj... [01:26:18] 6Labs, 10dev.wikimedia.org, 7Tracking: Create Labs project for "Data and developer hub" prototype - https://phabricator.wikimedia.org/T98275#1263684 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Done [01:26:19] 6Labs, 7Tracking: New Labs project requests (Tracking) - https://phabricator.wikimedia.org/T76375#1263687 (10yuvipanda) [01:45:03] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1403 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [01:56:34] Yuvipanda: Is there any API to connect to quarry.wmflabs.org? Or how do I connect a Django app to the Wikilabs DB? [02:04:10] yuvipanda: thanks for the fast turnaround! Do you give n00bs a next step after closing the task, like "Visit https://wikitech.wikimedia.org/wiki/Nova_Resource:Developer-doc and add documentation"? There seems to be nothing on wikitech that associates the new project with T98275 [02:15:05] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [02:20:17] 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org: https://wikitech.wikimedia.org/wiki/Special:Properties returns status 500 and 0-length page - https://phabricator.wikimedia.org/T98284#1263730 (10Spage) 3NEW [02:46:24] 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org: https://wikitech.wikimedia.org/wiki/Special:Properties returns status 500 and 0-length page - https://phabricator.wikimedia.org/T98284#1263754 (10Spage) As a workaround, https://wikitech.wikimedia.org/w/index.php?title=Special%3APrefixIndex&pre... [05:29:20] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:34:04] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 767169 bytes in 4.150 second response time [06:36:05] PROBLEM - Puppet failure on tools-bastion-02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:44:45] 6Labs, 10Beta-Cluster: Move logs off NFS on beta - https://phabricator.wikimedia.org/T98289#1263912 (10yuvipanda) 3NEW [06:45:47] 6Labs, 10Labs-Infrastructure, 3ToolLabs-Goals-Q4: Limit NFS bandwith per-instance - https://phabricator.wikimedia.org/T98048#1263919 (10yuvipanda) p:5High>3Unbreak! Happened again (T98289). I don't think manually hunting down things and feeling outraged is a good use of Opsen time. Setting priority appr... [06:46:06] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1403 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [07:01:04] RECOVERY - Puppet failure on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:11:05] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [07:46:33] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:51:02] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 767106 bytes in 2.788 second response time [08:48:47] 6Labs, 10Labs-Infrastructure, 5Continuous-Integration-Isolation: Include Base::Standard-packages in labs images - https://phabricator.wikimedia.org/T94995#1264068 (10hashar) When an instance boot (Trusty in this case: Notice: Finished catalog run in 56.05 seconds Cloud-init v. 0.7.5 running 'modules... [08:54:25] PROBLEM - Puppet staleness on tools-mailrelay-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [11:02:07] 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org: https://wikitech.wikimedia.org/wiki/Special:Properties returns status 500 and 0-length page - https://phabricator.wikimedia.org/T98284#1264391 (10Krenair) `PHP Fatal error: Call to undefined function wfViewPrevNext() in /srv/mediawik... [11:29:28] 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org: https://wikitech.wikimedia.org/wiki/Special:Properties returns status 500 and 0-length page - https://phabricator.wikimedia.org/T98284#1264436 (10Krenair) Caused by https://gerrit.wikimedia.org/r/#/c/147040/ [11:36:04] PROBLEM - Puppet failure on tools-static-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:43:27] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:44:01] PROBLEM - Puppet failure on tools-static-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:48:03] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 767106 bytes in 3.404 second response time [11:59:31] 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org, 5Patch-For-Review: https://wikitech.wikimedia.org/wiki/Special:Properties returns status 500 and 0-length page - https://phabricator.wikimedia.org/T98284#1264526 (10Krenair) a:3Krenair [12:13:58] RECOVERY - Puppet failure on tools-static-02 is OK: OK: Less than 1.00% above the threshold [0.0] [12:21:05] RECOVERY - Puppet failure on tools-static-01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:05:11] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:07:12] Coren: do you know what’s up with ^ ? There’ve been a few of those in the last couple of hours. [14:07:55] andrewbogott: Yeah, labstore is bursty with bouts of usage pegged at 100% [14:08:10] andrewbogott: I'm about to deploy trafic shaping in the hopes it will help a bit. [14:08:17] Coren: do you think that’s causing the intermittent puppet failures as well? [14:08:44] andrewbogott: I haven't seen correlation - those happen even when the nfs is (relatively) idle. [14:08:55] ok [14:08:59] puppet, as a rule, goes nowhere near NFS [14:09:08] It's all local disk. [14:09:21] Yeah, although it probably checks that the mounts are there… [14:09:35] Hm, which would probably succeed even if the mounts were frozen. So, nevermind [14:09:46] andrewbogott: It checks that they are in mtab. [14:10:02] yeah, definitely no nfs interaction there. [14:10:03] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 767106 bytes in 3.008 second response time [14:11:36] The problem with the puppet failures is that it's not reproductible. If you see one fail and go right away to the instance to do a puppet run by hand, it basically always works. :-( [14:13:33] Coren: yeah. I looked in virt1000 syslog for correlating issues and couldn’t find any. [14:13:53] I think I’m going to increae the puppet interval today just to see if reduced load on virt1000 stops the problem. [14:14:35] What's it at now? 30m right? [14:16:58] 20 I think. [14:19:54] !log maps maps-tiles3 has traffic shaping on - observing [14:19:58] Logged the message, Master [14:20:50] 6Labs, 10Beta-Cluster: Move logs off NFS on beta - https://phabricator.wikimedia.org/T98289#1264824 (10hashar) There are several topics: syslog -------- The instances have rsyslog configured to relay logs to deployment-bastion.eqiad.wmflabs The deployment-bastion is setup with `role::syslog::centralserver`... [14:25:43] Coren, again? :P [14:26:30] MaxSem: No, it's not out of line atm but because it's the biggest consumer of NFS right now it's also the best guinea pig. :-) [14:40:34] Hm. There are so many instances getting traffic through it's hard to see the effect of just one, even if it's pretty busy. [14:43:41] 6Labs, 10Beta-Cluster: Move logs off NFS on beta - https://phabricator.wikimedia.org/T98289#1264931 (10bd808) It should be pretty easy to get all of the logging in the beta cluster to flow into logstash. Apache and HHVM should be configured to log to syslog and then rsyslog rules used to forward to Logstash. T... [15:13:41] 6Labs: Discussion: can I park WikiSpy under a separate, simpler domain? - https://phabricator.wikimedia.org/T97846#1265010 (10d33tah) Five days passed and there was no discussion. Should I move this to the mailing list? [15:14:29] 6Labs: Request to review privacy policy and rules - https://phabricator.wikimedia.org/T97844#1265026 (10d33tah) A quick ping - I posted this five days ago and got no response. If noone speaks out until tomorrow, I will make a mailing list post. [15:40:06] 6Labs: Create WikiSpy project - https://phabricator.wikimedia.org/T96512#1265138 (10d33tah) Note to admins: my question from the previous post is still not answered. Is it the wrong place for such requests? [15:46:12] 6Labs: Discussion: can I park WikiSpy under a separate, simpler domain? - https://phabricator.wikimedia.org/T97846#1265155 (10Andrew) @d33tah, I think there's no discussion because nobody knows. I'll try to muster up some opinions. [15:47:02] andrewbogott: thanks [15:49:32] d33tah: sorry if I’m being dense… https://phabricator.wikimedia.org/T97844 asks us to review the privacy policy, but… where is that policy? [15:51:21] 6Labs, 6WMF-Legal: Discussion: can I park WikiSpy under a separate, simpler domain? - https://phabricator.wikimedia.org/T97846#1265168 (10Andrew) [15:51:22] 6Labs, 6WMF-Legal: Request to review privacy policy and rules - https://phabricator.wikimedia.org/T97844#1265170 (10Andrew) [15:51:55] 6Labs, 6WMF-Legal: Request to review privacy policy and rules - https://phabricator.wikimedia.org/T97844#1253240 (10Andrew) @d33tah, can you please attach said policy, or provide a link? [15:54:19] 6Labs, 6WMF-Legal: Request to review privacy policy and rules - https://phabricator.wikimedia.org/T97844#1265174 (10d33tah) Ah, sorry - it's there: http://wikispy.wmflabs.org/privacy http://wikispy.wmflabs.org/rules [16:01:04] 6Labs, 6WMF-Legal: Discussion: can I park WikiSpy under a separate, simpler domain? - https://phabricator.wikimedia.org/T97846#1265195 (10Andrew) @d33tah, we'll see if Legal chimes in, but here's my quick opinion: There's nothing in the Terms of Use that proscribes your doing this as long as - your branding... [16:01:22] Coren, can you please comment on https://phabricator.wikimedia.org/T97846 as well? [16:01:31] * Coren looks at it [16:01:46] 6Labs, 6WMF-Legal: Request to review privacy policy and rules - https://phabricator.wikimedia.org/T97844#1265200 (10jeremyb-phone) What does "additional terms" mean? What statistics do you have in mind? Can you run your stats, aggregation, etc. regularly and then toss the raw input when those have been genera... [16:06:39] 6Labs, 6WMF-Legal: Request to review privacy policy and rules - https://phabricator.wikimedia.org/T97844#1265211 (10d33tah) Unless somebody suggests me how can I implement the current functionality without keeping the IPs forever, I think that I would have to. As for additional terms - this is copied from en.w... [16:08:18] 6Labs, 6WMF-Legal: Discussion: can I park WikiSpy under a separate, simpler domain? - https://phabricator.wikimedia.org/T97846#1265212 (10d33tah) The thing is that I would need admins' support - I would need to park my domain on the labs servers, so I would like to get it officially OKed before I buy it. In o... [16:10:01] 6Labs, 6WMF-Legal: Discussion: can I park WikiSpy under a separate, simpler domain? - https://phabricator.wikimedia.org/T97846#1265220 (10coren) I agree there is nothing in the TOS that prevents it (provided Legal has no objection), but I should point out that we have no support for doing so within Labs, nor d... [16:13:20] 6Labs, 6WMF-Legal: Discussion: can I park WikiSpy under a separate, simpler domain? - https://phabricator.wikimedia.org/T97846#1265228 (10d33tah) I understand that this means that I cannot park the domain on your proxy and ask you to reconfigure it? Would it work then if I requested a separate IP and used the... [16:13:28] 6Labs, 6WMF-Legal: Request to review privacy policy and rules - https://phabricator.wikimedia.org/T97844#1265229 (10coren) You'll need to be //considerably// more precise in your privacy policy. You need to enumerate precisely what data you collect, how long you keep it, and how you will be using it. In addi... [16:15:00] 6Labs, 6WMF-Legal: Discussion: can I park WikiSpy under a separate, simpler domain? - https://phabricator.wikimedia.org/T97846#1265245 (10coren) Technologically, that would work. There are possible legal issues, however, so I'm going to add @slaporte in this task's CC so that he can chime in. [16:16:33] 6Labs, 6WMF-Legal: Request to review privacy policy and rules - https://phabricator.wikimedia.org/T97844#1265250 (10d33tah) Thanks coren! I will update the privacy policy to explain that (can I keep the IP list indefinitely?) and modify my program to ask whether it is okay to track unique views. Once I do that... [16:42:01] 6Labs, 6WMF-Legal: Request to review privacy policy and rules - https://phabricator.wikimedia.org/T97844#1265487 (10Ricordisamoa) >>! In T97844#1265211, @d33tah wrote: > As for the statistics, right now I am recording the IPs in the database in order to mark who watched which article in order to count unique v... [16:47:48] 6Labs, 6WMF-Legal: Request to review privacy policy and rules - https://phabricator.wikimedia.org/T97844#1265517 (10d33tah) In a way they are. I wanted to avoid authentication etc., but at the same time I plan to show a lot of potentially useless output to the users, so I need some way to tell interesting cont... [16:53:59] 6Labs, 6WMF-Legal: Request to review privacy policy and rules - https://phabricator.wikimedia.org/T97844#1265558 (10jeremyb-phone) I still don't understand exactly why/how you're using IPs or making logs unique. So therefore also don't see why people would abuse it or even what abuse would mean, how that would... [17:04:13] 6Labs, 10Labs-Infrastructure, 6operations, 7Ipv6: Enable ipv6 on labs - https://phabricator.wikimedia.org/T37947#1265579 (10scfc) [17:10:47] 6Labs, 10Labs-Infrastructure, 6operations, 7Ipv6: Enable IPv6 on wikitech.wikimedia.org - https://phabricator.wikimedia.org/T73218#1265628 (10scfc) [17:20:18] 6Labs, 6WMF-Legal: Request to review privacy policy and rules - https://phabricator.wikimedia.org/T97844#1265706 (10d33tah) Here's my thinking: I want a system that lets me separate interesting content from unimportant one. I assume that some of the things users will find will be unconvenient to third parties... [17:42:18] 6Labs, 6WMF-Legal: Request to review privacy policy and rules - https://phabricator.wikimedia.org/T97844#1265857 (10jeremyb) ok, so your content is partly crowdsourced. some providers make it very easy to cycle through IPs so that's not a silver bullet for abuse. I've been thinking over the last few comments... [17:50:41] 6Labs, 6WMF-Legal: Request to review privacy policy and rules - https://phabricator.wikimedia.org/T97844#1265888 (10d33tah) I know it's not a silver bullet, but it would definitely make any attacks more difficult. I make it easy to replicate my project in order to create separate websites. I do not plan to sh... [17:59:04] 6Labs, 6WMF-Legal: Request to review privacy policy and rules - https://phabricator.wikimedia.org/T97844#1265938 (10jeremyb) what about your historical RDNS DB? anyway, re SLA, part of that means that we generally don't host sites that will end up getting slashdotted. and I get the impression that you are ex... [18:01:54] 6Labs, 6WMF-Legal: Request to review privacy policy and rules - https://phabricator.wikimedia.org/T97844#1265952 (10d33tah) Yes, the last version of the website got 10k unique views within two days - I assumed that this is okay. The rDNS DB is taken from Solar7 research, at most I could share the rDNS queries... [18:16:42] 6Labs, 6WMF-Legal: Request to review privacy policy and rules - https://phabricator.wikimedia.org/T97844#1266003 (10d33tah) Does that mean I should be looking for a new hosting then? [18:18:00] 6Labs, 6WMF-Legal: Request to review privacy policy and rules - https://phabricator.wikimedia.org/T97844#1266008 (10jeremyb) >>! In T97844#1265952, @d33tah wrote: > Yes, the last version of the website got 10k unique views within two days - I assumed that this is okay. so then what if your site was broken for... [18:18:06] 6Labs, 10OpenStreetMap: Enable OSM Postgress machine access in labs - https://phabricator.wikimedia.org/T98382#1266009 (10Yurik) 3NEW [18:18:22] 6Labs, 10OpenStreetMap: Enable OSM Postgress machine access in labs - https://phabricator.wikimedia.org/T98382#1266016 (10Yurik) [18:19:52] 6Labs, 6WMF-Legal: Request to review privacy policy and rules - https://phabricator.wikimedia.org/T97844#1266022 (10jeremyb) >>! In T97844#1265952, @d33tah wrote: > The rDNS DB is taken from Solar7 research, at most I could share the rDNS queries for edits I discovered using the IRC bot. could you expand on t... [18:25:58] 6Labs, 6WMF-Legal: Request to review privacy policy and rules - https://phabricator.wikimedia.org/T97844#1266092 (10d33tah) >>! In T97844#1266008, @jeremyb wrote: >>>! In T97844#1265952, @d33tah wrote: >> Yes, the last version of the website got 10k unique views within two days > - I assumed that this is okay.... [19:18:53] !log ceph deleting project and instances. [19:18:58] Logged the message, dummy [19:27:34] Coren: re-writing the status page is fun [19:27:47] I am starting to HATE XML [19:27:58] Heh. Visible elements tends to be. :-) [19:28:23] Most fun dev: robotics. [19:29:05] !log titleblacklist deleting instance and project [19:29:05] titleblacklist is not a valid project. [19:29:11] oh, I suppose not :) [19:29:26] Coren: if qstat output json things would be nice and very easy [19:33:58] !log outreach deleting project and instance. [19:34:02] Logged the message, dummy [19:35:13] !log openstack-wiki deleting project and instance [19:35:19] Logged the message, dummy [19:38:14] !log metavidwiki deleting instance and project [19:38:18] Logged the message, dummy [19:39:07] !log glass deleting project and instance. [19:39:11] Logged the message, dummy [19:41:03] !log jawiki deleting project and instances; project seems to be abandoned. [19:41:16] Logged the message, dummy [19:42:42] !log greensmw deleting project and instance; abandoned. [19:42:45] Logged the message, dummy [19:43:18] 6Labs: allow routing between labs instances and public labs ips - https://phabricator.wikimedia.org/T96924#1266387 (10akosiaris) I 've looked into the specified change. It is about limiting the SNAT feature to a specified set of destination ranges. The behavior I described above it the result of the DNAT rules a... [19:45:00] !log documentation deleting project and instance; abandoned [19:45:04] Logged the message, dummy [19:47:35] yuvipanda: is the ‘discourse’ project still good for something or should I wipe it out? [19:48:16] andrewbogott: What's your qualification for "abandonned" in this case? [19:48:33] halfak: same question — is the ‘persistence’ project with instance ‘diffengine’ still useful or can I clean it up? [19:48:42] Coren: If no one has touched it since the migration from tampa. [19:49:12] I’m cleaning up things that have been on the ‘mothballed’ list since then and the instances are all still in the shutoff state. [19:49:34] Seems entirely reasonable. [19:50:40] Yeah, there’s been a few warnings about this, spaced months apart. I don’t think anyone is still left to care :) [19:51:47] !log category-sorting deleting instance and project; abandoned. [19:51:52] Logged the message, dummy [19:54:01] !log pediapress deleting project and instance; abandoned. [19:54:05] Logged the message, dummy [20:06:44] andrewbogott: wipe discourse [20:06:51] yuvipanda: ok! Thanks [20:11:58] !log phabricator Setup phab-02 with a "real" certificate (signed and validated) [20:12:02] Logged the message, Master [20:12:11] twentyafterfour: ^ now phab-02 shouldn't even give you security errors :) [20:29:31] Negative24: woah waoh. did you pay for a real cert? [20:29:43] no [20:30:06] Negative24: startssl? [20:30:25] its free for open source projects [20:31:21] aaah [20:31:23] nice nice :D [20:32:41] now I have to write puppet :( [20:32:51] (not related to ssl) [20:48:41] Coren: you around? [20:48:50] Betacommand: What be up? [20:48:54] we need to shrink mw-core. Phab has been going for 2hrs and is at 57% :D [20:49:06] Negative24: remember to not put it on NFS [20:49:25] yep [20:49:44] Negative24: :) [20:49:46] Coren: Im working on the python version ?status and noticed that glamtools is being a fairly major hog [20:50:03] Define 'hog' in context? [20:50:44] They set max VMEM 5.8GB for tasks that use less than 500mb. and have several copies running at the same time [20:51:56] Coren: 26 copies to be exact [20:52:16] o_O [20:52:36] I'm guessing 26 is a bug. [20:53:13] Coren: I would call that several bugs then [20:53:25] I'll be emailing. [20:53:44] setting 5.8G for something that uses 350mb [20:57:32] yuvipanda: how does ops configure ssl so that it isn't overwritten by puppet? wildcard imports in dir? [20:57:57] Coren: Ive got 80% of the tool ported to python, which is good news [20:58:57] Just CPU and VMEM data is left to fill in, then throw it all in a table [21:04:09] Make sure you keep the visual identity. We actualy had brandon carefully tweak that one. :-) [21:04:32] 6Labs: allow routing between labs instances and public labs ips - https://phabricator.wikimedia.org/T96924#1266814 (10Andrew) I've referred this to the openstack mailing list. I predict that a thousand voices cry out as one and tell me to install Neutron. [22:00:52] how do I purge files that aren't removed after removing a puppet role? [22:03:53] Negative24: once the files aren’t puppetized anymore they’re just files. There’s not any cohoerent system for cleaning up... [22:03:58] Other than starting with a fresh machine :) [22:07:50] chasemp: phab's puppet is having errors because of https://github.com/wikimedia/operations-puppet/blob/aed0ca8a00e5ae2046a409017ab9f4e52f12ba9e/modules/phabricator/manifests/init.pp#L217 and the tags are named "2015-05-06/1" [22:08:03] so its treating it as a directory [22:08:11] andrewbogott: ah very well [22:08:25] Negative24: yes I know, have some writeup to do from todays upgrade [22:08:36] long story short, using / in a tag name is just complicated [22:08:49] but it will be up to releng what they want to do [22:09:05] hmm do we have a bug report? [22:10:51] no but feel free to open one and I will link to it from my conclusions from the upgrade [22:10:57] it's matter of time not desire [22:11:31] what does the lock do? I could dig in the code but... [22:11:51] is this phab specific? [22:18:51] lock says don't upgrade teh local repo until this file is removd [22:19:06] i.e. we can merge a change and do it purposefully instead of randomly with next puppet run [22:20:19] chasemp: https://phabricator.wikimedia.org/T98411 [22:21:02] k [22:21:06] is that lock for phab_update_tag [22:23:11] that's a weird way to ask the question, the lock file is to lock a repo at a specific checkout [22:23:19] the script knows to remote it prior to upgrade [22:23:32] remove even [22:26:36] ah [22:41:09] anyways thanks for making the task :)