[00:47:55] *.wmflabs.org certificate expired :/ [00:48:28] RapidSSL SHA256 CA - G3 < > Expires on 9/15/2015 [00:49:36] effing https :/ [00:50:04] YuviPanda, how do i update that? [00:53:38] anyone? [00:53:47] thedj? [00:55:44] i'm from fiji so it expired first for me..it will follow for you all.. [00:57:46] all tools on "https" failing :/ [01:02:09] 6Labs, 10Tool-Labs: tools.wmflabs.org https certificate expired - https://phabricator.wikimedia.org/T112608#1640082 (10Bamyers99) 3NEW [01:02:44] thx bamyers [01:02:49] ouch https [01:03:10] only 30 min. dead, though [01:03:13] oops [01:03:22] i still effing hate https.. hate that some tools force 'https' :/ [01:04:02] i hate https as well because puppet usually installs apache with it enabled and sometimes I can't have it [01:04:10] someone forgot to pay the bill? :P [01:04:34] subscriptions* :P [01:04:47] no phab-02 is hosted differently. Its not behind the proxy so it only can go through http [01:04:50] 6Labs, 10Tool-Labs: *.wmflabs.org https certificate expired - https://phabricator.wikimedia.org/T112608#1640101 (10yuvipanda) p:5Triage>3Unbreak! [01:12:22] 6Labs, 10Tool-Labs: *.wmflabs.org https certificate expired - https://phabricator.wikimedia.org/T112608#1640110 (10yuvipanda) Looks like it might take us a day or so to renew them. Ugh. [01:12:46] :O [01:12:51] noooooooooooooooooooooooooooooooooooooooooooooooooooooooooo [01:19:31] 6Labs, 10Tool-Labs: *.wmflabs.org https certificate expired - https://phabricator.wikimedia.org/T112608#1640113 (10Negative24) Now, how will we remember to do this in the future? [01:20:09] 6Labs, 10Tool-Labs: *.wmflabs.org https certificate expired - https://phabricator.wikimedia.org/T112608#1640115 (10yuvipanda) I think there's a calendar somewhere with expiry dates, and this cert was just missed. I'll verify and make sure there's one such calendar. [01:27:25] 6Labs, 10Tool-Labs: *.wmflabs.org https certificate expired - https://phabricator.wikimedia.org/T112608#1640122 (10Krenair) T112542 refers to a 'tracking calendar'. And this is sort of part of that. [01:39:44] 6Labs, 10Tool-Labs: *.wmflabs.org https certificate expired (tools.wmflabs.org) - https://phabricator.wikimedia.org/T112608#1640139 (10Krinkle) [02:42:59] 6Labs, 10Tool-Labs: *.wmflabs.org https certificate expired (tools.wmflabs.org) - https://phabricator.wikimedia.org/T112608#1640256 (10Nicolas_Raoul) Thanks for working on this :-) Just for info, this breaks most Wikivoyage articles (they embed a dynamic map frame generated by tools.wmflabs.org): {F2604942} [02:50:10] 6Labs, 10Tool-Labs: *.wmflabs.org https certificate expired (tools.wmflabs.org) - https://phabricator.wikimedia.org/T112608#1640292 (10Negative24) Should the Wikivoyage tool perhaps be moved to a production or production-like machine to prevent such occurrences? From what I've heard, anything that breaks the a... [04:55:38] 6Labs, 3Labs-Sprint-114, 3labs-sprint-113: Evaluate gridengine's use of NFS and (possibly) move it to a different volume - https://phabricator.wikimedia.org/T111797#1640347 (10yuvipanda) I will admit to having absolutely zero confidence in GridEngine doing anything that involves NFS the way you'd expect it t... [05:10:49] 6Labs, 10Tool-Labs, 3Labs-Sprint-114, 5Patch-For-Review: Move toollabs off labsdebrepo into aptly - https://phabricator.wikimedia.org/T111708#1640356 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Done. [05:16:30] 6Labs, 3Labs-Sprint-107, 3Labs-Sprint-108, 3Labs-sprint-112: Evaluate a 'cluster solution' for use on Tool Labs - https://phabricator.wikimedia.org/T106475#1640364 (10yuvipanda) [05:16:31] 6Labs, 3Labs-Sprint-107, 3Labs-Sprint-108, 3Labs-Sprint-109, and 3 others: Evaluate kubernetes for use on Tool Labs - https://phabricator.wikimedia.org/T107993#1640361 (10yuvipanda) 5Open>3Resolved a:3yuvipanda We've decided to use Kubernetes. Announcement coming soon :) [05:16:42] 6Labs, 3Labs-Sprint-107, 3Labs-Sprint-108, 3Labs-sprint-112: Evaluate a 'cluster solution' for use on Tool Labs - https://phabricator.wikimedia.org/T106475#1469900 (10yuvipanda) We have decided to go with Kubernetes. Announcement coming soon :) [05:18:31] wow. certificate epired ? [05:18:34] yikes [05:42:15] 6Labs, 10Tool-Labs: *.wmflabs.org https certificate expired (tools.wmflabs.org) - https://phabricator.wikimedia.org/T112608#1640381 (10Wrh2) >>! In T112608#1640292, @Negative24 wrote: > Should the Wikivoyage tool perhaps be moved to a production or production-like machine to prevent such occurrences? From what... [05:49:21] thedj: yeah terrible :( [05:50:31] 6Labs, 10Tool-Labs: *.wmflabs.org https certificate expired (tools.wmflabs.org) - https://phabricator.wikimedia.org/T112608#1640400 (10yuvipanda) We are waiting on the CA to re-issue the certificate and will be back online as soon as that happens. Might be up to midday west coast time however :( [07:15:27] YuviPanda : there? [07:16:01] When I make a curl request to localhost:8080/w from within the vagrant machine, I don't get any output [07:16:04] Is that normal? [07:38:45] likely already known, but the certificate for toollabs has expired [10:13:30] Hi, how do I restart parsoid from within vagrant without running vagrant provision? [10:14:01] Because whenever I run vagrant provision, the localsettings.js gets overwritten [10:18:17] sitic: Hey! I'm facing a lot of lag in crosshatch lately. :( https://tools.wmflabs.org/crosswatch/ Over the past couple of days, that is. Any idea what's wrong? [11:08:45] hello, the /mnt partition vdb on the maps-warper instance was showing EXT3-fs errors, and yesterday when I wrote to it, was turned read only. I have rebooted it but now it's not showing up [11:09:40] would this be a phabricator issue? [11:14:32] chippy: it could indicate a hardware issue or an issue with the infra, so yes, please create a bug [11:15:13] valhallasw`cloud: okay will do, thanks [11:41:30] 6Labs, 6Discovery, 10Maps: maps-warper /mnt vbd partition errored, turned read only and went missing after reboot - https://phabricator.wikimedia.org/T112641#1641050 (10Chippyy) 3NEW [11:42:48] 6Labs, 10Maps: maps-warper /mnt vbd partition errored, turned read only and went missing after reboot - https://phabricator.wikimedia.org/T112641#1641063 (10Chippyy) [11:43:33] 6Labs, 10Maps: maps-warper /mnt vbd partition errored, turned read only and went missing after reboot - https://phabricator.wikimedia.org/T112641#1641050 (10Chippyy) [11:43:42] okay that should do it [11:57:22] Hi, there, is there a way to send an announcement to all Labs users and/or specifically to Tool Labs users? [11:57:34] It's an invitation to the Wikimedia Developer Summit [12:44:07] qgil, would the labs/tools list not be enough? [12:54:36] jynus, hi, I don't know. If the most active developers are subscribed to that one, then yes, it would be enough [13:00:56] qgil, I think they do, it is where maintenance is send (but it is a general discussion list), but do not know details about subscription (if it is strongly encouraged to all of them) [13:02:12] maybe Yuvi can give you a better alternative that I may not know [13:02:28] (when he comes back online) [13:02:44] 6Labs, 10Tool-Labs: Add a monitoring check for *.wmflabs.org certificate - https://phabricator.wikimedia.org/T112645#1641229 (10hashar) 3NEW [13:05:45] 6Labs, 10Tool-Labs, 7Icinga: Add a monitoring check for *.wmflabs.org certificate - https://phabricator.wikimedia.org/T112645#1641260 (10Florian) [13:06:53] Thank jynus , yes, I'm waiting for the sun to raise in the Pacific coast [13:07:15] another thing is whether YuviPanda's internal clock has anything to do with that sun, but whatever :) [13:08:05] he actually went to bed a few hours ago, so :-) [13:12:59] :) [13:18:48] qgil: yes, you can get a list of addresses from ldap, but I'm not sure if we /should/. The 'clean' option would be to send messages via wikitech, I think [13:20:25] qgil: maybe just using massmessage? [13:30:02] 6Labs, 3Labs-Sprint-107, 3Labs-Sprint-108, 3Labs-Sprint-109, and 3 others: Evaluate kubernetes for use on Tool Labs - https://phabricator.wikimedia.org/T107993#1641326 (10Negative24) @Yuvipanda Exciting! I'll be interested in how this plays out with my mediawiki-docker project. [13:44:20] I don't want to spam people, I just want to make sure that potential participants of online and face to face discussions are aware that we want to invite them to the Summit as well [13:46:43] i think you should just use the labs list [13:47:04] sending mails to all labs users using their labs mail addresses would not be appropriate here [13:53:23] ok, then I can just subscribe and send [13:53:32] will do later, need to go now [14:20:29] YuviPanda (or anyone): How do I talk to postgres from python? [14:21:19] andrewbogott: psycopg2 is the standard package for that, I think [14:21:45] but I've only used it from Django, so I'm not sure how to use it on labs [14:22:12] http://initd.org/psycopg/docs/usage.html [14:25:28] 6Labs, 10Tool-Labs, 7Icinga: Add a monitoring check for *.wmflabs.org certificate - https://phabricator.wikimedia.org/T112645#1641491 (10JohnLewis) There is an SSL check (though not for this apparently) which reports warning 60 before expiry and then 30 days before. This really should be added in a meta way. [14:26:27] hm, doesn’t look like psycopg2 is installed on tools [14:33:45] this can be fixed ;-) [14:35:33] (or just install it in a virtualenv?) [14:37:09] andrewbogott: pip install psycopg2 should compile it without issues [14:38:53] I don’t suppose you know if/where a random users creds are for postgres? [14:40:23] andrewbogott: I'm not even sure if we have general user accounts on postgres. For testing, you can use the osm user, I think [14:40:31] https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Database#Configuring_PGAdmin_for_OSM_access [14:42:00] hm, no, that's not passwordless [14:42:06] That’ll work, although I don’t like the idea of alerts getting tripped by osm changes [15:02:58] announcement, announcement! [15:14:03] Is the https sertificate thing being fixed? [15:15:34] Base: see the topic [15:26:43] 6Labs, 5Patch-For-Review: Setup an availability checker for all labsdb hosts - https://phabricator.wikimedia.org/T107449#1641697 (10Andrew) url tests are up for 1001, 1002, 1003, 1005. I'm still unclear about how to handle postgres. [15:28:58] 6Labs, 3Labs-Sprint-114, 3labs-sprint-113: Evaluate gridengine's use of NFS and (possibly) move it to a different volume - https://phabricator.wikimedia.org/T111797#1641713 (10scfc) (If you think that T109362 is related to NFS, please explain there.) I agree with both your points. This would be an expensiv... [16:06:59] 6Labs, 10Tool-Labs: *.wmflabs.org https certificate expired (tools.wmflabs.org) - https://phabricator.wikimedia.org/T112608#1641865 (10coren) 5Open>3Resolved a:3coren The new certificates have been issued, and I've pushed them to the servers (specifically, the labs-wide dynamicproxy and the tools-specifi... [16:08:02] 6Labs, 10Labs-Infrastructure: Switch to a multi_host nova network - https://phabricator.wikimedia.org/T107731#1641875 (10mark) [16:08:03] 6Labs, 10Labs-Infrastructure: Enable arp on the labs vlan - https://phabricator.wikimedia.org/T108581#1641873 (10mark) 5Open>3declined We had decided not to move forward with this. [16:11:00] All Labs HTTPS endpoints should now be using the new certificate. [16:11:56] Thanks coren [16:13:25] great, a certificate. thanks coren [16:50:14] 6Labs, 6operations, 10ops-eqiad, 3Labs-Sprint-114, 3ToolLabs-Goals-Q4: Make certain ports and cables between the labstores and shelves are numbered/named and labeled, and make sure that the diagram(s) reflect that. - https://phabricator.wikimedia.org/T112549#1642029 (10coren) @cmjohnson: what I need is a... [17:04:37] 6Labs, 7Puppet: Move all labs-only puppet roles to manifests/role/labs - https://phabricator.wikimedia.org/T107167#1642060 (10JanZerebecki) Updated https://wikitech.wikimedia.org/wiki/Puppet_coding#Roles to reflect this. [17:32:54] 6Labs, 6operations, 10ops-eqiad, 3Labs-Sprint-114, 3ToolLabs-Goals-Q4: Make certain ports and cables between the labstores and shelves are numbered/named and labeled, and make sure that the diagram(s) reflect that. - https://phabricator.wikimedia.org/T112549#1642206 (10Cmjohnson) labstore1002 port 0 to l... [19:03:32] YuviPanda: I could use guidance about how to test the postgres labsdb — right now a lack a user/pass [19:04:09] andrewbogott: oh we need to manually create one for you... [19:04:16] Postgres is still opt in only [19:04:31] andrewbogott: I'm just heading for lunch can I create it when I'm.back? [19:04:45] sure, any time. Thanks [19:09:29] 6Labs, 10Labs-Infrastructure: Switch to a multi_host nova network - https://phabricator.wikimedia.org/T107731#1642667 (10Andrew) 5Open>3declined [20:29:01] 6Labs, 10Tool-Labs: *.wmflabs.org https certificate expired (tools.wmflabs.org) - https://phabricator.wikimedia.org/T112608#1642956 (10Wrh2) >>! In T112608#1641865, @coren wrote: > The new certificates have been issued, and I've pushed them to the servers (specifically, the labs-wide dynamicproxy and the tools... [20:36:34] !log dumps suspending dumps-1 briefly to assess its performance impact [20:36:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Dumps/SAL, dummy [20:39:44] (03PS1) 10Jforrester: [WIP] Support wmf branches like '1.23.0-wmf.6' [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/238542 (https://phabricator.wikimedia.org/T107192) [20:40:48] !log dumps resuming dumps-1 [20:40:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Dumps/SAL, dummy [20:52:29] !log mwoffliner trying to rescue mwoffliner3 which is stuck in ‘reboot’ state [20:52:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mwoffliner/SAL, dummy [20:52:49] 6Labs, 10Tool-Labs: Packages from the aptly server are not installable - https://phabricator.wikimedia.org/T112699#1643006 (10scfc) 3NEW [20:57:17] (03Abandoned) 10Ejegg: Filter more WMDE projects from #wm-fundraising [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/231002 (owner: 10Ejegg) [20:58:09] 6Labs, 10Tool-Labs: Packages from the aptly server are not installable - https://phabricator.wikimedia.org/T112699#1643031 (10valhallasw) Alternatively, we can sign packages in aptly and add that GPG key as trusted key using apt::repository::keyfile. [21:57:05] mehrrit-wm: hello [22:13:43] (03PS1) 10Yuvipanda: Log to stdout only [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/238619 [22:20:04] (03PS1) 10Yuvipanda: Add simple Dockerfile [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/238622 [22:22:54] 6Labs, 5Patch-For-Review: Setup an availability checker for all labsdb hosts - https://phabricator.wikimedia.org/T107449#1643321 (10Andrew) Yuvi, can haz postgres creds? [23:52:34] [13flask-mwoauth] 15robinkrahl opened pull request #14: Update link to OAuth consumer registration form (06master...06patch-1) 02http://git.io/vZbmC