[00:00:09] RECOVERY - Puppet run on tools-worker-1010 is OK: OK: Less than 1.00% above the threshold [0.0] [00:00:58] doctaxon: dr = running, but trying to delete it [00:02:04] deletion seems to be not possible anyhow [00:11:17] 06Labs, 10Tool-Labs: Deletion of job fails - https://phabricator.wikimedia.org/T135927#2315529 (10doctaxon) [00:15:00] 06Labs, 10Tool-Labs: Deletion of job fails - https://phabricator.wikimedia.org/T135927#2315542 (10doctaxon) [00:26:00] PROBLEM - Puppet run on tools-worker-1010 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [01:01:07] RECOVERY - Puppet run on tools-worker-1010 is OK: OK: Less than 1.00% above the threshold [0.0] [03:16:28] !log tools.taxonbot Force deleted job 5737063 (T135927) [03:18:59] 06Labs, 10Tool-Labs, 15User-bd808: Deletion of job fails - https://phabricator.wikimedia.org/T135927#2315589 (10bd808) 05Open>03Resolved a:03bd808 I force deleted the job which was stuck in "deletion" state. [03:30:43] !log tools.stashbot Restarted to apply fix for https://github.com/bd808/tools-stashbot/issues/10 [03:30:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL, Master [03:34:49] 06Labs, 10Tool-Labs, 13Patch-For-Review, 15User-bd808: PHP 5.5 sessionclean cron job hanging on tool labs bastions - https://phabricator.wikimedia.org/T135861#2315594 (10bd808) 05Open>03Resolved [03:54:48] 06Labs, 10Tool-Labs: "webservice" command (with jsub) doesn't work in crontab - https://phabricator.wikimedia.org/T135348#2315597 (10bd808) This is at least somewhat expected. The job runner grid nodes (where jsub/qsub jobs run) are not configured to allow them to submit other jobs to the grid. The would expla... [04:07:15] 06Labs, 10Tool-Labs: Increase webservice memory limit for piagetbot - https://phabricator.wikimedia.org/T135346#2295936 (10bd808) What do you think you need the limit to be raised to? The default limit is 4G. Note for self (and others): The memory limit for a given webservice can be raised by creating a file... [04:08:48] 10Tool-Labs-tools-Other: Provide a list of the videos available on video2commons servers - https://phabricator.wikimedia.org/T134914#2315604 (10bd808) [04:33:21] PROBLEM - Puppet run on tools-worker-1003 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [04:37:11] 06Labs, 10Tool-Labs, 10Pywikibot-core: ToolLabs git clone is really slow - https://phabricator.wikimedia.org/T134232#2258764 (10bd808) >>! In T134232#2258879, @AbdealiJK wrote: > I finally got mediawiki-core to clone successfully. I times it using the `time` command and here is the data: > ``` > real 173m37.... [04:45:32] 06Labs, 10Tool-Labs, 10Pywikibot-core: ToolLabs git clone is really slow - https://phabricator.wikimedia.org/T134232#2315637 (10AbdealiJK) @bd808 That is awesome ! It would be useful to add this to the Gotchas section - https://wikitech.wikimedia.org/wiki/Help:Tool_Labs#Gotchas [04:47:07] AbdealiJK: it's a wiki ;) [04:47:25] Will add it :) [04:58:09] 06Labs, 10Tool-Labs, 10Pywikibot-core: ToolLabs git clone is really slow - https://phabricator.wikimedia.org/T134232#2315642 (10AbdealiJK) 05Open>03Resolved a:03AbdealiJK I've added the suggestion mentioned by @bd808 on the wiki - https://wikitech.wikimedia.org/w/index.php?title=Help%3ATool_Labs&type=r... [05:05:48] 06Labs, 10Tool-Labs: /home/ium missing replica.my.cnf - https://phabricator.wikimedia.org/T134066#2315657 (10bd808) [05:06:21] 06Labs, 10Tool-Labs, 10Pywikibot-core: Tool Labs: shared Pywikibot code not available - https://phabricator.wikimedia.org/T125505#2315658 (10Ato_01) 05Resolved>03Open p:05Triage>03Normal [05:06:23] 06Labs, 10Tool-Labs: /home/tanvir missing replica.my.cnf - https://phabricator.wikimedia.org/T131546#2315660 (10bd808) [05:11:15] 06Labs, 10Tool-Labs, 07Tracking: [Tracking] Tool Labs users missing replica.my.cnf - https://phabricator.wikimedia.org/T135931#2315661 (10bd808) [05:11:43] 06Labs, 10Tool-Labs: /home/ium missing replica.my.cnf - https://phabricator.wikimedia.org/T134066#2315675 (10bd808) [05:11:45] 06Labs, 10Tool-Labs: /home/tanvir missing replica.my.cnf - https://phabricator.wikimedia.org/T131546#2315676 (10bd808) [05:11:47] 06Labs, 10Tool-Labs, 07Tracking: [Tracking] Tool Labs users missing replica.my.cnf - https://phabricator.wikimedia.org/T135931#2315674 (10bd808) [05:12:47] 06Labs, 10Tool-Labs, 07Tracking: [Tracking] Tool Labs users missing replica.my.cnf - https://phabricator.wikimedia.org/T135931#2315661 (10bd808) [05:14:53] 06Labs, 10Tool-Labs, 10DBA: p50380g50816__pop_stats (popularpages) using 53G on labsdb1001 (enwiki) - https://phabricator.wikimedia.org/T133326#2315678 (10bd808) @Mr.Z-man / @kaldari Any update on this question? Is the table active, and if so is there any way to reduce its footprint? [05:19:01] RECOVERY - Puppet run on tools-worker-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [06:32:04] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Not Available - string 'Magnus' not found on 'http://tools.wmflabs.org:80/' - 530 bytes in 0.093 second response time [06:57:04] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 3669 bytes in 0.016 second response time [08:22:06] 06Labs, 10Tool-Labs: Increase webservice memory limit for piagetbot - https://phabricator.wikimedia.org/T135346#2315741 (10tom29739) I should have thought about 6GB would be enough for it. [08:34:51] PROBLEM - Host tools-bastion-01 is DOWN: CRITICAL - Host Unreachable (10.68.17.228) [08:57:46] RECOVERY - Puppet run on tools-webgrid-lighttpd-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [09:26:00] PROBLEM - Puppet run on tools-worker-1010 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [10:54:22] Hi, shared pywikibot instance is missing. In /shared/pywikipedia/core there is nothing. Who can fix it? I also tried to run PWB from core_old, then I got nameerror in my user-config.py (NameError: name 'user_home_path' is not defined). User with root access can read my config file from /home/urbanecm/.pywikibot/user-config.py in labs, or I can send it by mail. [10:56:02] (fix, there is pywikibot directory and version file in it but no runable scripts present) [10:57:41] there is package mwparserfromhell too with some python scripts) [11:13:12] 06Labs, 10Tool-Labs, 10Pywikibot-core: Tool Labs: shared Pywikibot code not available - https://phabricator.wikimedia.org/T125505#1989468 (10russblau) This has been happening multiple times per month, sometimes more than once in a week. When it happens, it can be fixed simply by re-running the nightly job on... [11:40:15] 06Labs, 10Tool-Labs, 10Pywikibot-core: Tool Labs: shared Pywikibot code not available - https://phabricator.wikimedia.org/T125505#2315915 (10Urbanecm) p:05Normal>03High No PWB scripts currently present, stopping run of all PWB-based scripts, at least high priority I think. [11:43:37] 06Labs, 10Tool-Labs, 10Pywikibot-core: Tool Labs: shared Pywikibot code not available - https://phabricator.wikimedia.org/T125505#1989468 (10jayvdb) Do we know what the cause is yet? [12:04:22] PROBLEM - Puppet run on tools-worker-1003 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [12:39:29] RECOVERY - Puppet run on tools-worker-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [13:30:31] 10Quarry: Add a possibility to delete a draft - https://phabricator.wikimedia.org/T135908#2314983 (10Halfak) I'm not sure we should delete anything (short of doxing and the like). Maybe we can "archive" a draft so that it doesn't appear in your history of queries unless a "show archived queries" box is checked.... [13:59:11] 10Quarry: Add a possibility to delete a draft - https://phabricator.wikimedia.org/T135908#2316003 (10Dvorapa) That's not so different from my current solution of renaming the unused queries to "~" and use them as a clean paper for new queries in the future. [14:01:19] 10Quarry: Add a possibility to delete a draft - https://phabricator.wikimedia.org/T135908#2316006 (10Dvorapa) I understand not deleting already published queries, but (in my opinion) drafts could be deleted without loosing any important content [14:05:29] 10Quarry: Add a possibility to delete a draft - https://phabricator.wikimedia.org/T135908#2316008 (10yuvipanda) Indeed, everything is public, and I think we're careful to not call them 'drafts' (if not we should be!)- published just means you are finished tweaking them for a bit. Not sure what the right thing... [14:44:51] Hi Guys - Did the standard for database names change? Having an issue creating a new one under the tools.cluebotng user on tools-db/tools.labsdb. [14:49:33] Damianz_web, what name do you want? [14:50:40] there were some months ago an issue with database names allowing user_* databases, when the only supported name was user__database (too underscores) [14:55:14] Access denied for user 's52585'@'%' to database 's52585__interface' < Is what I'm trying currently. It's for a test install so ideally need to create+drop etc a few times, not fussed about the name so much. [14:55:43] strange, that should be allowed [14:55:48] let me check [14:58:40] Damianz_web, it seems there is something wrong with the grants, has this user (tool) been created recently? [15:01:35] It was created at least a year ago, just not used :) [15:02:10] I can fix this for you, but could you file a ticket meanwhile with the issue while I do it? [15:02:21] you are not the only user affected [15:02:24] Sure :) And thanks [15:03:54] the one time fix is easy but the long-term fix will take a bit more [15:05:05] I do not see a current s52585__interface db, however [15:06:17] 06Labs, 10Tool-Labs, 10Pywikibot-core: Tool Labs: shared Pywikibot code not available - https://phabricator.wikimedia.org/T125505#2316116 (10valhallasw) I think the main issue is a combination of delete-then-clone plus slow NFS. It's not entirely clear to me whether the script fails halfway or whether it's j... [15:06:24] 06Labs, 10Labs-Infrastructure: Unable to create database on tools.labsdb - https://phabricator.wikimedia.org/T135947#2316117 (10DamianZaremba) [15:07:20] Should be there now, creation works :) [15:09:17] 06Labs, 10Labs-Infrastructure, 10DBA: Unable to create database on tools.labsdb - https://phabricator.wikimedia.org/T135947#2316136 (10jcrespo) It seems that s52585 (now manually fixed) and other users lack the provileges ON `s52585\_\_%`.* TO 's52585'@'%' privileges. I can fix this, but @yuvipanda, could... [15:09:26] 06Labs, 10Labs-Infrastructure, 10DBA: Unable to create database on tools.labsdb - https://phabricator.wikimedia.org/T135947#2316138 (10jcrespo) p:05Triage>03Low [15:09:48] 06Labs, 10Labs-Infrastructure, 10DBA: Missing grants on tools.labsdb - https://phabricator.wikimedia.org/T135947#2316117 (10jcrespo) [15:16:25] 06Labs, 10Tool-Labs, 10Pywikibot-core: Tool Labs: shared Pywikibot code not available - https://phabricator.wikimedia.org/T125505#2316144 (10zhuyifei1999) >>! In T125505#2315919, @jayvdb wrote: > Do we know what the cause is yet? T126666#2020483? [15:25:07] RECOVERY - Puppet run on tools-worker-1010 is OK: OK: Less than 1.00% above the threshold [0.0] [15:32:52] 10Labs-Other-Projects, 06Discovery, 10Maps: Receiving warning emails about failed puppet runs - multiple instances - https://phabricator.wikimedia.org/T135942#2316196 (10dschwen) [15:44:34] 10Labs-Other-Projects, 06Discovery, 10Maps: Receiving warning emails about failed puppet runs - multiple instances - https://phabricator.wikimedia.org/T135942#2316203 (10dschwen) running manually it seems to hang after ``` Info: /Stage[main]/Standard::Ntp::Client/Ntp::Daemon[client]/Service[ntp]: Unschedul... [16:28:05] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Temporarily Unavailable - string 'Magnus' not found on 'http://tools.wmflabs.org:80/' - 383 bytes in 2.003 second response time [16:33:03] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 3669 bytes in 0.048 second response time [16:49:52] PROBLEM - Puppet staleness on tools-elastic-02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [43200.0] [17:06:18] PROBLEM - Puppet run on tools-docker-builder-03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:16:06] !log defragmenting db1028 [17:16:06] defragmenting is not a valid project. [17:23:25] 06Labs, 10Tool-Labs, 10Pywikibot-core: Tool Labs: shared Pywikibot code not available - https://phabricator.wikimedia.org/T125505#2316293 (10valhallasw) I have [[ https://github.com/pywikibot/Pywikibot-nightly-creator/commit/d0192941b267a98b26021d6d1d55b7c0d6e14086 | rewritten parts of the the nightly code]]... [17:37:24] 10Labs-Other-Projects, 06Discovery, 10Maps: Receiving warning emails about failed puppet runs - multiple instances - https://phabricator.wikimedia.org/T135942#2316009 (10notconfusing) I'm receiving the same issue on with `Alert: puppet failed on wigi.wikidumpparse.eqiad.wmflabs` . There's no indication on ho... [17:40:59] 06Labs, 10Tool-Labs: Increase webservice memory limit for piagetbot - https://phabricator.wikimedia.org/T135346#2316303 (10bd808) >>! In T135346#2315741, @tom29739 wrote: > I should have thought about 6GB would be enough for it. {{done}} ``` $ become admin $ echo 6G > /data/project/.system/config/piagetbot.we... [17:41:15] RECOVERY - Puppet run on tools-docker-builder-03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:42:37] 06Labs, 10Tool-Labs: Increase webservice memory limit for piagetbot - https://phabricator.wikimedia.org/T135346#2316304 (10tom29739) Thank you! >>! In T135346#2316303, @bd808 wrote: >>>! In T135346#2315741, @tom29739 wrote: >> I should have thought about 6GB would be enough for it. > > {{done}} > ``` > $ beco... [17:43:04] 06Labs, 10Tool-Labs: Increase webservice memory limit for piagetbot - https://phabricator.wikimedia.org/T135346#2316305 (10tom29739) 05Open>03Resolved a:03tom29739 [18:16:33] bd808: thanks for the ln suggestion :-) [18:17:08] 10Tool-Labs-tools-Other, 06Community-Tech, 07I18n: Add jQuery i18n to Pageviews Analysis - https://phabricator.wikimedia.org/T133766#2316320 (10Purodha) @MusikAnimal Thank you. [18:19:17] Hi, any wikitech admin on? [18:19:29] White_Master: what's up? [18:19:46] valhallasw`cloud, mp [19:35:32] 06Labs, 10Tool-Labs, 07Tracking: Tool Labs users missing replica.my.cnf (tracking) - https://phabricator.wikimedia.org/T135931#2316382 (10Danny_B) [20:05:31] 06Labs, 10Tool-Labs, 10Pywikibot-core: Tool Labs: shared Pywikibot code not available - https://phabricator.wikimedia.org/T125505#2316408 (10Ato_01) Since about 3 hours ago I can not access '/shared/pywikipedia/core' and I am receiving the following messages from my bot: python: can't open file '/shared/pywi... [20:10:56] 06Labs, 10Tool-Labs, 10Pywikibot-core: Tool Labs: shared Pywikibot code not available - https://phabricator.wikimedia.org/T125505#2316409 (10valhallasw) There was indeed a permissions mixup on /data/project/pywikibot/public_html, which should now also be fixed... [20:18:07] 06Labs, 10Tool-Labs, 10Pywikibot-core: Tool Labs: shared Pywikibot code not available - https://phabricator.wikimedia.org/T125505#2316412 (10Ato_01) Thank you. Let's see how does it work in the next couple of days. :) [20:26:34] 06Labs, 10Phabricator: Upgrade phab-01.wmflabs.org - https://phabricator.wikimedia.org/T127617#2316435 (10Danny_B) It would be seriously very handy to have actual version somewhere to test its features before using them here to prevent cluttering and possible issues... [20:40:11] 06Labs, 10Phabricator: Upgrade phab-01.wmflabs.org - https://phabricator.wikimedia.org/T127617#2316451 (10TerraCodes) Just wondering, why do we have 4+ phab installs on wmflabs.org? 1) phab-01.wmflabs.org 2) phab-02.wmflabs.org 3) phab-03.wmflabs.org 4) phab-04.wmflabs.org [21:03:12] Hi. I was wondering if I could please have my tools request reviewed? :) [21:07:02] hi [21:07:34] error.log show nothing however I can't start my app https://tools.wmflabs.org/video2commons-test/ [21:18:47] "You have not enabled a web service for your tool, or it has stopped working because of a fatal error" [21:19:49] The_Photographer: maybe uwsgi.log contains relevant data? [21:25:05] Platonides, nothing important, only requests [21:25:30] maye a execution time problem [21:26:00] PROBLEM - Puppet run on tools-worker-1010 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [21:26:11] that's odd [21:26:44] mmm [21:30:54] whats the preferred whois tool for templates? /whois/ seems to be dead. [22:01:00] RECOVERY - Puppet run on tools-worker-1010 is OK: OK: Less than 1.00% above the threshold [0.0] [23:17:52] hi [23:18:25] someone could check why my tool is not runing, log show nothing [23:18:30] https://tools.wmflabs.org/video2commons-test/