[01:57:46] 10Labs-project-Wikistats, 13Patch-For-Review: Wikistats 2.2 [beta] gives internal server error 500 for all csv, ssv and xml formats - https://phabricator.wikimedia.org/T165879#3293758 (10Dzahn) Hi @Xqt CSV format works again, using new code, but sticking to the same format, outputs all columns it can find. N... [03:06:49] 10Labs-project-Wikistats, 13Patch-For-Review: Wikistats 2.2 [beta] gives internal server error 500 for all csv, ssv and xml formats - https://phabricator.wikimedia.org/T165879#3293771 (10Dzahn) Hi again, changed my mind about the SSV thing, that also works again now (URLs as before) after additional change ab... [05:57:00] 10Labs-project-Wikistats, 13Patch-For-Review: Wikistats 2.2 [beta] gives internal server error 500 for all csv, ssv and xml formats - https://phabricator.wikimedia.org/T165879#3293869 (10Xqt) I only use xml format for the bots. I'll wait a bit. Thanks a lot. [06:42:16] PROBLEM - Puppet errors on tools-exec-1432 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:42:18] PROBLEM - Puppet errors on tools-exec-1427 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [06:43:13] 06Labs, 10DBA: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3293932 (10jcrespo) @russblau No, new databases are in read only mode. We are unsure how we can support that and how (because of T156869), so for now all new servers are in full read only mode. You can create databases on to... [06:44:15] PROBLEM - Puppet errors on tools-exec-1412 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:17:16] RECOVERY - Puppet errors on tools-exec-1427 is OK: OK: Less than 1.00% above the threshold [0.0] [07:19:14] RECOVERY - Puppet errors on tools-exec-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [07:22:17] RECOVERY - Puppet errors on tools-exec-1432 is OK: OK: Less than 1.00% above the threshold [0.0] [13:14:55] Petscan is not qorking [13:14:57] *working [13:15:16] It's returning zero results for something I know should have at least 10 [13:37:48] hi, i've added an OAuthConsumerRegistration proposal, but now that i've submited it, i can't change some values, the callback url can be changed after aprouved? [13:41:55] alchimista: no, I don't think so [13:42:36] valhallasw`cloud, ok, i'll cancel the request and make a new one then [13:56:51] (03PS5) 10Jean-Frédéric: Proof of concept to harvest Wikidata into monuments database [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/354961 (https://phabricator.wikimedia.org/T165988) [15:15:36] bd808 are you registering the channel? Would i be able to rejoin once you do that please? [15:16:42] paladox: its registered with chanserv now, so you can hop back in whenever you want [15:16:49] Thanks :) [17:52:41] 06Labs: Request increased quota (floating ip) for getstated labs project - https://phabricator.wikimedia.org/T166324#3295255 (10Steinsplitter) [18:09:24] 06Labs: Request increased quota (floating ip) for getstarted labs project - https://phabricator.wikimedia.org/T166324#3295349 (10bd808) [19:42:32] 10Labs-project-Wikistats, 13Patch-For-Review: Wikistats 2.2 [beta] gives internal server error 500 for all csv, ssv and xml formats - https://phabricator.wikimedia.org/T165879#3295535 (10Dzahn) @Xqt I expected you to use the CSV for some reason. Didn't expect XML, does this mean you need it to be the exact sa... [19:50:46] 06Labs, 07Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#3295541 (10Andrew) [20:07:18] 06Labs, 10Labs-Infrastructure: labservices1002 slow puppet runs and IO issues - https://phabricator.wikimedia.org/T164675#3295567 (10Andrew) 05stalled>03Resolved [20:08:24] 06Labs: nova-fullstack is losing instances on creation - https://phabricator.wikimedia.org/T165555#3295574 (10Andrew) No drops since this bug was logged. [20:18:56] hello once more, i'm developing a python + flask +oauth web app to have some web controls on a bot. The app is runnig on the grid, but subprocess.Popen(['/qstat']) gives me "None". What's missing so that the app can get the qstat result of the tool account? [20:20:52] in other words, the subprocess.Popen(['qstat']) should give the same result as if qstat where emitted by my account on the bastion [20:21:29] that *should* work I think [20:21:40] alchimista: what's in the stderr output? [20:22:04] and what is the exact output you get with Popen? [20:22:49] huh. /usr/bin/qstat: No such file or directory [20:22:53] in any case, qstat is available on tools-webgrid hosts (not on tools-exec hosts) [20:22:58] that's on exec-1004 [20:23:58] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Write unit tests for Xtools - https://phabricator.wikimedia.org/T165400#3295625 (10DannyH) [20:24:18] valhallasw`cloud: it's installed but -- error: denied: host "tools-webgrid-lighttpd-1417.tools.eqiad.wmflabs" is neither submit nor admin host [20:24:27] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Update all of the Xtools with new backend - https://phabricator.wikimedia.org/T165399#3295626 (10DannyH) [20:24:41] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Unknown project should not cause an Internal Server Error - https://phabricator.wikimedia.org/T166332#3295627 (10DannyH) [20:24:43] that sounds like a grid misconfiguration -- iirc webgrid hosts are supposed to be able to submit jobs [20:24:54] true [20:25:29] * bd808 shakes fist at complexity of OGE config [20:26:05] well, i've made a take ~www/.../app, just in case, and restarted the webservice, seems to work for now [20:26:51] and using /usr/bin/qstat, just qstat wasn't working [20:28:25] !log tools sudo qconf -as tools-webgrid-lighttpd-1417.tools.eqiad.wmflabs [20:28:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:31:28] !log tools Added tools-webgrid-lighttpd-1412 and tools-webgrid-lighttpd-1413 as submit hosts [20:31:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:31:41] looks like we had 3 misconfigured [20:32:05] oh actually more than that :/ [20:32:44] !log tools Added tools-webgrid-lighttpd-14{19,2[0-8]} as submit hosts [20:32:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:34:05] valhallasw`cloud: I see a lot of files in /var/lib/gridengine/etc/exechosts for servers that are no longer on the grid. Do you think its safe to rm them? [20:34:40] I would suggests moving them rather than deleting, but otherwise I think it should be safe? [20:34:43] on the other hand, it's OGE [20:34:51] heh. yeah. [20:35:07] maybe not the thing to do late Friday before a 3 day weekend [20:35:14] I'll make a task [20:46:13] 06Labs, 10Tool-Labs: Cleanup stale files in /var/lib/gridengine/etc/exechosts - https://phabricator.wikimedia.org/T166412#3295652 (10bd808) [20:46:54] 06Labs, 10Tool-Labs: Cleanup stale files in /var/lib/gridengine/etc/exechosts - https://phabricator.wikimedia.org/T166412#3295665 (10bd808) There's nothing about cleaning these up in https://wikitech.wikimedia.org/wiki/Portal:Tool_Labs/Admin#Decommission_a_node [20:47:37] 06Labs, 10wikitech.wikimedia.org: Move wikitech-static to Chicago - https://phabricator.wikimedia.org/T164271#3295666 (10Andrew) Docs for how to do this are at https://support.rackspace.com/how-to/transferring-images-between-regions-of-the-rackspace-open-cloud/ [20:55:12] 06Labs, 10Tool-Labs: Cleanup stale files in /var/lib/gridengine/etc/exechosts - https://phabricator.wikimedia.org/T166412#3295689 (10bd808) @valhallasw and I //think// we can just move/delete these, but late in the day on a Friday before a 3-day weekend seems like a poor time to do this kind of cleanup. [21:43:34] https://lists.debian.org/debian-devel-announce/2017/05/msg00002.html [21:44:12] stretch is planed for stable release next month. june 17 [21:48:10] 06Labs, 10Labs-Infrastructure: Can't do git clone from some labs-instances - https://phabricator.wikimedia.org/T166170#3295762 (10Luke081515) Same for pushing: ``` STDERR Pushing to https://********@github.com/Luke081515/**** fatal: unable to access 'https://********@github.com/Luke081515/****/': Failed to co... [21:50:15] 06Labs, 10Labs-Infrastructure: Can't do git clone from some labs-instances - https://phabricator.wikimedia.org/T166170#3295766 (10Paladox) >>! In T166170#3295762, @Luke081515 wrote: > Same for pushing: > ``` > STDERR > Pushing to https://********@github.com/Luke081515/**** > fatal: unable to access 'https://*... [22:12:39] 06Labs, 10Labs-Infrastructure: Can't do git clone from some labs-instances - https://phabricator.wikimedia.org/T166170#3295789 (10Luke081515) Would be less work if I can fix this another way. What I found out too: You can push and pull from bitbucket for example, but for github, it does not work. [22:19:36] 06Labs, 10Labs-Infrastructure: Can't do git clone from some labs-instances - https://phabricator.wikimedia.org/T166170#3287386 (10bd808) url-downloader.wikimedia.org is the HTTP proxy that is used in the production network so that production servers can talk to a tightly controlled set of external hosts. An in... [23:44:44] 06Labs, 15User-bd808, 06cloud-services-team (Kanban): Consult with technical community on Cloud Services rebranding plan - https://phabricator.wikimedia.org/T165094#3295847 (10bd808)