[00:47:49] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [01:22:50] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [02:18:39] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [04:05:49] 06Labs, 10Horizon: Need Horizon dashboard for manipulating service groups - https://phabricator.wikimedia.org/T91989#1100569 (10AlexMonk-WMF) Would suggest this could go to Striker now, but tools wasn't the only project to use them: ```krenair@tools-bastion-03:~$ ldapsearch -x objectClass=posixgroup | grep dn:... [04:31:39] 10Striker, 07Epic: Manually connect existing Diffusion repos to tools in toolsadmin - https://phabricator.wikimedia.org/T144282#2607951 (10Krenair) [04:31:41] 10Striker: attach Diffusion repo TGFT to tools.giftbot - https://phabricator.wikimedia.org/T144130#2607948 (10Krenair) 05Open>03Resolved a:03Krenair You should see it on toolsadmin now [04:40:56] 10Striker: Please link some tools maintained by Legoktm to their Gerrit repos in toolsadmin - https://phabricator.wikimedia.org/T144258#2593577 (10Krenair) Tried to do the first one, got an error. Request ID 9a6f4826b14448c28cf39024b36c68bd Tool: wikibugs Name: labs-tools-wikibugs2 Phid: PHID-REPO-7gfle35moyddw2... [04:41:22] (03PS1) 10BryanDavis: Bump striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/308504 [04:41:39] 10Striker, 07Epic, 07Tracking: Manually connect existing Diffusion repos to tools in toolsadmin - https://phabricator.wikimedia.org/T144282#2607954 (10Krenair) [04:48:06] 06Labs: Request creation of wmwcourse labs project - https://phabricator.wikimedia.org/T144388#2598041 (10bd808) Small instances work fine for mw-vagrant at low request volume. [04:49:16] (03CR) 10BryanDavis: [C: 032] Bump striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/308504 (owner: 10BryanDavis) [04:49:22] (03Merged) 10jenkins-bot: Bump striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/308504 (owner: 10BryanDavis) [05:06:26] 10Striker: Please link some tools maintained by Legoktm to their Gerrit repos in toolsadmin - https://phabricator.wikimedia.org/T144258#2607974 (10bd808) >>! In T144258#2607952, @Krenair wrote: > Tried to do the first one, got an error. Request ID 9a6f4826b14448c28cf39024b36c68bd > Tool: wikibugs > Name: labs-to... [05:26:59] 10Striker: Create Wikitech/LDAP accounts via a new user friendly guided workflow - https://phabricator.wikimedia.org/T144710#2607979 (10bd808) [05:30:33] 10Striker: Allow management of LDAP SSH keys - https://phabricator.wikimedia.org/T144711#2607994 (10bd808) [05:35:37] 10Striker: Check for 2FA protection and enforce validation of 2FA tokens - https://phabricator.wikimedia.org/T144712#2608006 (10bd808) [05:36:49] 10Striker: Check for 2FA protection and enforce validation of 2FA tokens - https://phabricator.wikimedia.org/T144712#2608022 (10bd808) [05:36:51] 10Striker: Allow management of LDAP SSH keys - https://phabricator.wikimedia.org/T144711#2608021 (10bd808) [05:40:42] 10Striker: Add a web shell allowing people to perform actions as their tool from striker - https://phabricator.wikimedia.org/T144713#2608023 (10yuvipanda) [07:39:15] 06Labs: Request increased quota for reading-web-staging labs project - https://phabricator.wikimedia.org/T144623#2608127 (10phuedx) We can safely reading-web-staging-2. [07:40:51] PROBLEM - Free space - all mounts on tools-services-01 is CRITICAL: CRITICAL: tools.tools-services-01.diskspace.root.byte_percentfree (<10.00%) [08:49:54] !log tools.wikiloves Edited templates/eventmain.html on the server to avoid ZeroDivision error. [10:59:11] I'm a Labs user and I can't change my crontab (crontab -e, crontab crontab.txt), changes don't save [11:17:36] 06Labs, 10Labs-Infrastructure: I can't use crontab on Labs - https://phabricator.wikimedia.org/T144731#2608528 (10MaxBioHazard) [11:20:05] 06Labs, 10Labs-Infrastructure, 10Tool-Labs: I can't use crontab on Labs - https://phabricator.wikimedia.org/T144731#2608541 (10MaxBioHazard) [11:37:13] 06Labs, 10Labs-Infrastructure, 10Tool-Labs: I can't use crontab on Labs - https://phabricator.wikimedia.org/T144731#2608588 (10MaxBioHazard) https://www.youtube.com/watch?v=izQ3vmvWgYo video of the problem. [14:34:20] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Nodepool image metadata show: true is invalid/rejected by OpenStack API - https://phabricator.wikimedia.org/T144753#2609115 (10hashar) [14:36:30] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Nodepool image metadata show: true is invalid/rejected by OpenStack API: {u'show': u'true'} is not of type 'string' - https://phabricator.wikimedia.org/T144753#2609129 (10hashar) [14:47:01] 06Labs: Request creation of wmwcourse labs project - https://phabricator.wikimedia.org/T144388#2609158 (10Nikerabbit) Okay, I can use small instances then. [14:50:09] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Nodepool image metadata show: true is invalid/rejected by OpenStack API: {u'show': u'true'} is not of type 'string' - https://phabricator.wikimedia.org/T144753#2609168 (10hashar) On labnet1002 the API log shows the query has been done ag... [14:52:45] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Nodepool image metadata show: true is invalid/rejected by OpenStack API: {u'show': u'true'} is not of type 'string' - https://phabricator.wikimedia.org/T144753#2609195 (10hashar) [15:01:57] it seems running crontab -e / -r is having issues? [15:07:25] mafk: mbh reported a problem with cron (e [15:07:34] mafk: https://phabricator.wikimedia.org/T144731 [15:07:51] !log cron -e / -r apparently has some issue. See T144731 [15:07:52] T144731: I can't use crontab on Labs - https://phabricator.wikimedia.org/T144731 [15:07:53] cron is not a valid project. [15:07:59] !log labs cron -e / -r apparently has some issue. See T144731 [15:07:59] labs is not a valid project. [15:07:59] T144731: I can't use crontab on Labs - https://phabricator.wikimedia.org/T144731 [15:08:05] same issue [15:08:23] it probably applies to whole of tools [15:08:33] maybe you can highlight it on the labs mailling list [15:09:04] 06Labs, 10Labs-Infrastructure, 10Tool-Labs: I can't use crontab on Labs - https://phabricator.wikimedia.org/T144731#2608528 (10MarcoAurelio) I confirm I am having the same issue. [15:09:41] I'm too lazy to log-in to the mail to do so [15:11:06] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Nodepool image metadata show: true is invalid/rejected by OpenStack API: {u'show': u'true'} is not of type 'string' - https://phabricator.wikimedia.org/T144753#2609258 (10hashar) I got rid of the images properties via `glance image-updat... [15:11:59] hashar: now that you're here, is there any test on jenkins to test HTML? npm? [15:13:04] mafk: the way we have build the Jenkins jobs, they really just execute package managers install [15:13:13] and run a "test" command define in the repo [15:13:25] so for javascript, one typically add a package.json [15:13:31] that list all the dependencies needed for testing [15:13:38] then define the "test" entry point with something like: [15:13:43] jsonlint/jshint/jsduck? [15:13:54] "scripts": { "test": "my_html_checker ." } [15:14:02] and Jenkins will hapilly run: npm install && npm test [15:14:08] which should run the command you have listed in [15:14:32] there is a bunch of doc at https://www.mediawiki.org/wiki/Continuous_integration/Entry_points [15:14:43] with dummy examples to add jshint/jsonlint [15:15:18] and there is https://github.com/stylelint/stylelint :] [15:16:52] hashar: what I meant, is there any template in zuul.yaml that has those jobs? [15:16:59] yeah [15:17:00] npm [15:17:03] or node-jessie [15:17:07] cant remember the exact name [15:17:16] you will also need to add a proper package.json in your repo [15:17:28] with npm too? [15:17:32] yes [15:17:39] :/ [15:17:47] it's never easy heh [15:17:52] so [15:18:01] - name: foo/bar/your/repo/in/gerrit [15:18:03] template: [15:18:06] - name: npm [15:18:10] - name: jsduck [15:18:15] and that should covers it [15:18:21] once CI is set [15:18:29] the rest can be tweaked/ changed directly in your repo [15:18:34] I already have the php55 test set [15:22:08] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Nodepool image metadata show: true is invalid/rejected by OpenStack API: {u'show': u'true'} is not of type 'string' - https://phabricator.wikimedia.org/T144753#2609286 (10hashar) I have deleted the image and created a new one without usi... [15:40:13] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Nodepool image metadata show: true is invalid/rejected by OpenStack API: {u'show': u'true'} is not of type 'string' - https://phabricator.wikimedia.org/T144753#2609345 (10hashar) And I eventually remembered that Nodepool does inject prop... [15:44:36] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Nodepool image metadata show: true is invalid/rejected by OpenStack API: {u'show': u'true'} is not of type 'string' - https://phabricator.wikimedia.org/T144753#2609357 (10hashar) a:03hashar [15:48:57] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Joeygreen1 was created, changed by Joeygreen1 link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Joeygreen1 edit summary: Created page with "{{Tools Access Request |Justification=I wish to create applications that use the Wikimedia projects. |Completed=false |User Name=Joeygreen1 }}" [15:49:16] hashar: finally got to my mail and send the note as suggested (cron not working) [16:05:30] 06Labs, 10Labs-Infrastructure, 10Tool-Labs: I can't use crontab on Labs - https://phabricator.wikimedia.org/T144731#2608528 (10Andrew) Just so I'm clear -- the problem is with editing cron, not with the crons themselves firing, correct? [16:09:04] andrewbogott, seems so [16:09:07] I'm looking at it now [16:09:25] when saving a crontab as that user, it does this: tools.m+ 14955 0.0 0.0 40136 2868 pts/39 SN+ 16:08 0:00 | \_ /usr/bin/ssh tools-cron-01.tools.eqiad.wmflabs /usr/bin/crontab - [16:10:01] That cron tool is weird, maybe it uses nfs to distribute itself? [16:10:02] and seems to get stuck there [16:10:47] and the file begins... [16:10:48] #! /usr/bin/perl [16:10:51] (iirc tools has a custom-made crontab tool that overrides the normal tool in order to distribute crons across the project. No idea what it does and how it works…) [16:10:52] # Copyright � 2013 Marc-Andr� Pelletier [16:10:58] yep [16:11:07] well, I'm now finding out [16:11:20] so I don't have to, thank you! [16:14:51] in my tools' case it seems to get stuck even doing the ssh [16:17:23] something seems up with this tools-cron-01 host [16:21:08] can't do basic things without problems [16:24:31] can't even run top here [16:24:48] andrewbogott? [16:24:50] console log is empty [16:24:59] ok, looking [16:25:36] sometimes ssh to this host worked, sometimes it didn't. sometimes some commands here randomly hung for no reason [16:25:49] oom maybe [16:25:52] htop and top both sit there looking at you [16:25:54] I wonder what it would hurt to reboot it? [16:25:59] was wondering the same [16:26:01] what does 'free' say? [16:26:36] it doesn't seem to have a whole lot to say right now, that freezes too [16:26:44] ok, I'm rebooting it [16:27:34] !log tools rebooting tools-cron-01 because it is hanging all over the place [16:27:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, dummy [16:30:54] Krenair: ok, it's down and back… does the cron tool act any better now? [16:31:29] yes [16:31:32] much better [16:31:43] can edit my tools' crontab now [16:32:35] I can only hope that it also actually does things [16:37:58] 06Labs, 10Labs-Infrastructure, 10Tool-Labs: I can't use crontab on Labs - https://phabricator.wikimedia.org/T144731#2608528 (10Nettrom) SuggestBot's cron jobs for editing https://en.wikipedia.org/wiki/Wikipedia:Community_portal/Opentask appear to be executing just fine. I also noticed that changing crontab w... [16:38:13] andrewbogott, well syslog looks active with cron commands going through [16:38:20] ok, good [16:38:24] though it did before, so [16:38:27] I texted yuvi as well, for reassurance [16:38:43] Probably it hit a storm of too-many-crons-at-midnight and that will happen again in a day [16:40:06] 06Labs, 10Labs-Infrastructure, 10Tool-Labs: I can't use crontab on Labs - https://phabricator.wikimedia.org/T144731#2609452 (10Andrew) I restarted tools-cron-01 and that seems to have resolved the problem for the moment. /Probably/ there's an underlying cause, though (specifically, a broken cron job that do... [17:00:32] 10Tool-Labs-tools-Pageviews: Add 'offset' or pagination to Massviews - https://phabricator.wikimedia.org/T144760#2609465 (10MusikAnimal) [17:03:23] 10Tool-Labs-tools-Pageviews: Massviews should show first 500 pages of a category, not 200 - https://phabricator.wikimedia.org/T144761#2609480 (10MusikAnimal) [17:03:45] 10Tool-Labs-tools-Pageviews: Massviews should show first 500 pages of a category, not 200 - https://phabricator.wikimedia.org/T144761#2609494 (10MusikAnimal) [17:43:36] 10Tool-Labs-tools-wikiloves, 10Wiki-Loves-Monuments (2016): Error in links wikiloves statistics tool - https://phabricator.wikimedia.org/T144669#2609536 (10JeanFred) [17:44:53] 10Tool-Labs-tools-wikiloves: Create http://tools.wmflabs.org/wikiloves/monuments/2016 - https://phabricator.wikimedia.org/T144472#2600816 (10JeanFred) Indeed! :) I suggest everyone has a look to see if I did not miss any country, and that the start dates are correct ! [17:44:58] 10Tool-Labs-tools-wikiloves: Create http://tools.wmflabs.org/wikiloves/monuments/2016 - https://phabricator.wikimedia.org/T144472#2609539 (10JeanFred) 05Open>03Resolved a:03JeanFred [17:55:31] 10Tool-Labs-tools-wikiloves, 10Wiki-Loves-Monuments (2016): Error in links wikiloves statistics tool - https://phabricator.wikimedia.org/T144669#2609563 (10Danilo) 05Open>03Resolved [[https://github.com/ptwikis/wikiloves/commit/f71efdcdc008baab9a197fe9e420f032b15ec96e|Fixed]] [18:23:17] andrewbogott everything ok now? [18:32:56] 10Tool-Labs-tools-Pageviews, 06Analytics-Kanban: siteviews data for 2016 August 27 appears to be empty - https://phabricator.wikimedia.org/T144159#2609603 (10Nuria) 05Open>03Resolved [19:37:19] PROBLEM - Puppet run on tools-docker-builder-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:41:26] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/JogoBot was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=819471 edit summary: [19:44:23] RECOVERY - Host tools-secgroup-test-103 is UP: PING OK - Packet loss = 0%, RTA = 0.81 ms [19:53:20] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [20:21:14] RECOVERY - Host secgroup-lag-102 is UP: PING OK - Packet loss = 0%, RTA = 0.49 ms [20:27:01] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [20:35:56] RECOVERY - Host tools-secgroup-test-102 is UP: PING OK - Packet loss = 0%, RTA = 0.48 ms [20:47:25] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [20:58:19] 06Labs, 10Labs-Infrastructure, 10Tool-Labs: I can't use crontab on Labs - https://phabricator.wikimedia.org/T144731#2609763 (10MaxBioHazard) 05Open>03Resolved a:03MaxBioHazard Yes, works for me now. [21:13:55] 06Labs, 10Labs-Infrastructure, 10Tool-Labs: I can't use crontab on Labs - https://phabricator.wikimedia.org/T144731#2609776 (10AlexMonk-WMF) 05Resolved>03Open a:05MaxBioHazard>03None As @Andrew wrote this is being kept open for a while because we should check on it again later, it may re-occur [21:45:56] 10Striker, 07Epic, 07Tracking: Manually connect existing Diffusion repos to tools in toolsadmin - https://phabricator.wikimedia.org/T144282#2609825 (10bd808) [21:45:58] 10Striker, 15User-bd808: Please link some tools maintained by Legoktm to their Gerrit repos in toolsadmin - https://phabricator.wikimedia.org/T144258#2609822 (10bd808) 05Open>03Resolved a:03bd808 * https://toolsadmin.wikimedia.org/tools/id/wikibugs/repos/id/labs-tools-wikibugs2 * https://toolsadmin.wikim... [21:48:20] bd808: thanks :) [22:14:59] legoktm: yw [22:38:43] (03PS1) 10BryanDavis: Improve contrib-admin display of goals and repos [labs/striker] - 10https://gerrit.wikimedia.org/r/308667 [22:39:29] (03CR) 10jenkins-bot: [V: 04-1] Improve contrib-admin display of goals and repos [labs/striker] - 10https://gerrit.wikimedia.org/r/308667 (owner: 10BryanDavis) [22:40:24] (03PS2) 10BryanDavis: Improve contrib-admin display of goals and repos [labs/striker] - 10https://gerrit.wikimedia.org/r/308667 [23:28:25] !log tools.jouncebot Added jouncebot.sh script borrowed from stashbot to stop, start, restart, rotate logs, tail logs, ... [23:28:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.jouncebot/SAL, Master [23:28:56] !log tools.jouncebot Cherry-picked https://gerrit.wikimedia.org/r/#/c/308087/ for testing [23:29:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.jouncebot/SAL, Master [23:29:15] !log tools.jouncebot Cherry-picked https://gerrit.wikimedia.org/r/#/c/308086/ for testing [23:29:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.jouncebot/SAL, Master [23:31:33] 06Labs, 10Wikimedia-Site-requests, 10wikitech.wikimedia.org, 13Patch-For-Review: Enable math extension on wikitech - https://phabricator.wikimedia.org/T126338#2610006 (10Dereckson) [23:31:36] 06Labs, 10Wikimedia-Site-requests, 10wikitech.wikimedia.org, 13Patch-For-Review: Allow wikitech to write files for Math - https://phabricator.wikimedia.org/T126628#2610004 (10Dereckson) 05Open>03Resolved a:03Dereckson [23:31:55] 06Labs, 10Wikimedia-Site-requests, 10wikitech.wikimedia.org, 13Patch-For-Review: Allow wikitech to write files for Math - https://phabricator.wikimedia.org/T126628#2019112 (10Dereckson) Extension now successfully write files to /srv/math-images. [23:33:41] 06Labs, 06Operations, 10Wikimedia-Site-requests, 10wikitech.wikimedia.org, 13Patch-For-Review: Enable math extension on wikitech - https://phabricator.wikimedia.org/T126338#2610008 (10Dereckson) Math extension can now write files successfully in /srv/math-images folder. Last piece of the configuration i... [23:59:47] 10Tool-Labs-tools-Pageviews: Show error massage when requesting unsupported date ranges - https://phabricator.wikimedia.org/T144782#2610038 (10MusikAnimal)