[00:37:29] 10MediaWiki-extensions-OpenStackManager, 13Patch-For-Review, 05WMF-deploy-2016-10-11_(1.28.0-wmf.22): Update HTMLForm definitions to use `'dropdown' => true` rather than `'cssclass' => 'mw-chosen'` - https://phabricator.wikimedia.org/T143445#2702026 (10Krenair) 05Open>03Resolved a:03Paladox [01:57:22] PROBLEM - Puppet staleness on tools-worker-1005 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [03:52:38] 06Labs, 10wikitech.wikimedia.org, 13Patch-For-Review: mwscriptwikiset broken when using all.dblist on terbium - https://phabricator.wikimedia.org/T132383#2702166 (10Krenair) [03:57:09] PROBLEM - Puppet run on tools-webgrid-lighttpd-1208 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [04:37:07] RECOVERY - Puppet run on tools-webgrid-lighttpd-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [04:48:06] hi, is there a way to get notified (by mail?) that a job submitted with jsub was finished? [08:33:05] 06Labs, 10Tool-Labs: puppet failure on tools-worker-1005.tools.eqiad.wmflabs - https://phabricator.wikimedia.org/T147672#2702250 (10yuvipanda) 05Open>03Resolved a:03yuvipanda I've re-enabled it - was disabling it temporarily to do a demo at the ops offsite. I think in the long run, puppet shouldn't be di... [09:07:23] RECOVERY - Puppet staleness on tools-worker-1005 is OK: OK: Less than 1.00% above the threshold [3600.0] [12:47:36] (03PS1) 10Lokal Profil: [NOT TESTED] Add Nigeria in English to the database [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/314860 (https://phabricator.wikimedia.org/T143573) [12:54:23] (03CR) 10Lokal Profil: "So me updating Ubuntu meant that tox is now failing when I run it locally (see below). Funnily one of the few tests to pass was test_monum" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/314860 (https://phabricator.wikimedia.org/T143573) (owner: 10Lokal Profil) [12:56:55] (03CR) 10Jean-Frédéric: "Thanks for the patch :)" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/314860 (https://phabricator.wikimedia.org/T143573) (owner: 10Lokal Profil) [13:04:12] (03CR) 10Lokal Profil: "> > > OK. When I try to follow the steps I get the following error" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313452 (owner: 10Jean-Frédéric) [13:32:50] (03CR) 10Lokal Profil: Expand ReadMe on development environment (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313452 (owner: 10Jean-Frédéric) [13:35:51] (03PS2) 10Lokal Profil: Add Nigeria in English to the database [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/314860 (https://phabricator.wikimedia.org/T143573) [13:37:44] (03CR) 10Lokal Profil: "> Thanks for the patch :)" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/314860 (https://phabricator.wikimedia.org/T143573) (owner: 10Lokal Profil) [14:31:07] (03PS7) 10Lokal Profil: Only output one primkey warning per page [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/312975 (https://phabricator.wikimedia.org/T138633) [14:32:12] (03CR) 10Lokal Profil: "I dropped the test and opened a ticket at T147752 to track a possible solution in pywikibot." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/312975 (https://phabricator.wikimedia.org/T138633) (owner: 10Lokal Profil) [14:41:53] (03CR) 10Lokal Profil: "> > Flake8: "F821 undefined name 'unicode'"" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/314860 (https://phabricator.wikimedia.org/T143573) (owner: 10Lokal Profil) [14:45:20] (03Abandoned) 10Lokal Profil: [NOT READY]Require lat, lon, image, monument_article and registrant_url [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309862 (https://phabricator.wikimedia.org/T55813) (owner: 10Lokal Profil) [14:45:27] (03Abandoned) 10Lokal Profil: [Not READY] Require lat, lon, image, monument_article, registrant_url [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309859 (https://phabricator.wikimedia.org/T55813) (owner: 10Lokal Profil) [15:55:23] why is /public/dumps not available on my new instance? [15:56:41] Nikerabbit: NFS is only available if explicitly enabled, but iirc that is per-project and not per-instance [15:56:57] Nikerabbit: might just be a puppet thing, where it takes a few runs for everything to get set correctly [15:57:35] valhallasw`cloud: I have not explicitly enabled such thing, never saw anything like that [15:57:42] besides all documentation claims it is there by default [15:58:36] I guess I'll do without [15:59:34] Nikerabbit: https://wikitech.wikimedia.org/wiki/Help:Shared_storage#.2Fpublic.2Fdumps "there are a few shared storage directories that can be made available on request." [16:11:52] valhallasw`cloud: is /data also among the "few"? [17:16:10] Nikerabbit: /data/project and /data/scratch are on NFS [17:31:04] hi there [17:31:07] PROBLEM - SSH on tools-webgrid-lighttpd-1209 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:35:57] RECOVERY - SSH on tools-webgrid-lighttpd-1209 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [17:38:41] freddy2001: hello [17:46:06] i have a problem with jsub and php. i can run my script without any problems on tools-bastion-03 but if i want to run it as a cron i get a php parse error on tools-cron-01 [17:47:44] the problematic line in my script is: return ["title" => $content, "level" => $sectionlevel]; [17:47:55] Nikerabbit: the longer explanation of dumps and other NFS mounts not being enabled in Labs projects by default any more is that they were a common source of outages due to the core NFS servers being overloaded. It is possible to add hiera configuration to re-enable some or all NFS mounts for a project. I'll check to see if we have clear docs on that. [17:48:11] * bd808 assumes we do not [17:48:50] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:49:20] the line for this cron in crontab -e is: 45 16 * * * /usr/bin/jsub -N cron-tools.freddy2001-JWPaktuell -l release=trusty -once -quiet php -f /data/project/freddy2001/LarusBot/JWPaktuell.php [17:49:55] krenair@tools-cron-01:~$ php --version [17:49:56] PHP 5.5.9-1ubuntu4.20 (cli) (built: Oct 3 2016 13:00:37) [17:50:17] freddy2001: add "-l release=trusty" to your jsub command [17:50:37] confusingly the bastions are running a newer version of PHP than the current default job runners [17:50:40] oh, jsub in the cron. right [17:50:52] we will be fixing that versy soon -- https://wikitech.wikimedia.org/wiki/Tools_Precise_deprecation [17:50:55] should've read it all [17:51:21] week after next, nice [17:51:31] oh... but I see -l release-trust in your pasted cmd line.... [17:51:43] hmmmm [17:53:29] what if you make it print the result of gethostname, then eval("['a' => true]") ? [17:55:03] yeah, having the exec node name may help us guess what's going wrong [17:56:47] 5.5.9: tools-bastion-[02-03,05].tools.eqiad.wmflabs,tools-checker-[01-02].tools.eqiad.wmflabs,tools-cron-01.tools.eqiad.wmflabs,tools-exec-[1401-1410].tools.eqiad.wmflabs,tools-webgrid-generic-[1401-1404].tools.eqiad.wmflabs,tools-webgrid-lighttpd-[1401-1416,1418].tools.eqiad.wmflabs (37) [17:56:54] 5.3.10: tools-exec-[1201-1221].tools.eqiad.wmflabs,tools-exec-gift.tools.eqiad.wmflabs,tools-precise-dev.tools.eqiad.wmflabs,tools-webgrid-lighttpd-[1201-1210].tools.eqiad.wmflabs (33) [17:57:37] tools-services-01.tools.eqiad.wmflabs also has 5.5.9 but without xdebug [18:01:14] i think it is tools-cron-01.tools.equiad.wmflabs [18:02:17] freddy2001: are you trying to run the command there from an ssh session, or are you just getting an email or error log from when cron tries to run your script? [18:03:13] it is just in the error log [18:03:39] 10Tool-Labs-tools-Matthewrbowker's-tools: Typo at webpage - https://phabricator.wikimedia.org/T147758#2702686 (10Luke081515) [18:04:30] *nod* so that seems to imply that the "-l release=trusty" argument to jsub is not working as expected in that command. The parse error would be because the job is running with PHP 5.3 on a precise host I think. [18:05:01] or if i submit it with mail() at the beginning, "Received: from tools.freddy2001 by tools-cron-01.tools.eqiad.wmflabs with local (Exim 4.82) (envelope-from )" is in the header [18:07:54] bd808: yeah the docs could be more updated (I fixed one place)) [18:11:28] freddy2001: your cron tab entry isn't what you think it is. It is missing the "-l release=trusty" that is needed. [18:12:51] this looks like some sort of replication problem? On tools-bastion-02.tools.eqiad.wmflabs it is what you think it is, but on tools-cron-01.tools.eqiad.wmflabs the crontab is different [18:13:19] Krenair: do you know what voodoo pushes the actual crontab files from the bastions to the cron host? [18:13:22] 45 16 * * * /usr/bin/jsub -N cron-tools.freddy2001-JWPaktuell -l release=trusty -once -quiet php -f /data/project/freddy2001/LarusBot/JWPaktuell.php <- isnt it in it? [18:13:30] look at the crontab source [18:13:35] * Krenair is a bit busy right now [18:13:49] bd808: crontab sshs to tools-cron and ribs crontab there [18:13:54] Runs* [18:15:35] hmmm... curious. I am definately seeing different output from `crontab -l` from the tools-dev vs cron-01 [18:18:52] That's definitely not right. Maybe something odd with $PATH? [18:20:09] root@tools-cron-01:~# crontab -u tools.freddy2001 -l | grep release [18:20:09] 16 18 * * * jsub -N cron-tools.freddy2001-JWPaktuell -l release=trusty -once -quiet "php -f /data/project/freddy2001/LarusBot/JWPaktuell.php" [18:23:50] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:25:43] there is -l release=trusty in it, Krenair [18:25:47] yes [18:27:04] it changed. I have scrollback showing different data: "09 18 * * * jsub -N cron-tools.freddy2001-JWPaktuell -once -quiet php -f /data/project/freddy2001/LarusBot/JWPaktuell.php" [18:27:24] but it does look right now, so I'd expect it to work tmorrow [18:29:40] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Prepare and check production and labs-side filtering for olowiki - https://phabricator.wikimedia.org/T147302#2702752 (10MarcoAurelio) Since the wiki has been created and is now live, is this resolved? [18:30:04] okay, so i'll try it tomorrow again [18:30:54] * bd808 crosses fingers [18:33:56] !log tools removed empty local crontabs for {yuvipanda, yuvipanda, tools.toolschecker} on {tools-webgrid-lighttpd-1404, tools-webgrid-lighttpd-1204, tools-checker-01}. No other local crontabs remaining. [18:34:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [18:34:36] bd808: given that there are no local crontabs, I'm really confused why you would see two different values :/ [18:34:48] other than 'someone was editing them at the same time' [18:36:08] I have no good explanation [18:36:14] let's blame perl :) [19:05:52] ^ good solution [19:13:27] xD [19:37:19] PROBLEM - Puppet run on tools-docker-builder-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:00:26] PROBLEM - Puppet run on bdsync-deb is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:07:50] 06Labs, 10Tool-Labs, 10Pywikibot-core: New pages are not being created by pagefromfile.py - https://phabricator.wikimedia.org/T147766#2702908 (10Blahma)