[00:17:08] 6Labs, 10Tool-Labs: toolsbeta-puppetmaster3 can't resolve hiera('labs_puppet_master') - https://phabricator.wikimedia.org/T106627#1473655 (10scfc) No, it's not fixed. `/etc/puppet/hiera.yaml` gets overwritten with the contents of `modules/puppetmaster/files/labs.hiera.yaml` which doesn't reference a Hiera fil... [00:36:20] 6Labs, 7Mobile, 3Mobile-App-Sprint-62-Android-Summer-Breeze, 3Wikipedia-Android-App: Decide what to do with the android-build machine - https://phabricator.wikimedia.org/T104190#1473721 (10bearND) @yuvipanda. Thank you. It works! :) [01:18:15] 6Labs, 10Tool-Labs, 5Patch-For-Review: toolsbeta-puppetmaster3 can't resolve hiera('labs_puppet_master') - https://phabricator.wikimedia.org/T106627#1473828 (10scfc) a:3scfc Tested the patch on Toolsbeta. [01:20:21] 6Labs, 7Puppet: Puppet Trebuchet provider compares refname with commit sha1 and does NOT refresh the git repo! - https://phabricator.wikimedia.org/T77002#1473832 (10ori) @Hashar is this still an issue? [02:33:01] 6Labs, 7Mobile, 3Mobile-App-Sprint-62-Android-Summer-Breeze, 3Wikipedia-Android-App: Decide what to do with the android-build machine - https://phabricator.wikimedia.org/T104190#1473901 (10yuvipanda) 5Open>3Resolved Cool :) [08:51:55] <_acs_> re [09:03:33] 6Labs, 10Tool-Labs, 10Wikimania-Hackathon-2015, 15User-Bd808-Test: Create template PHP application for use on Tool Labs based on Slim, Twig and Wikimedia libraries - https://phabricator.wikimedia.org/T90092#1474240 (10Aklapper) What is the status of this task, now that Wikimania 2015 is over? As this task... [09:08:34] 6Labs, 10Tool-Labs, 10Wikimania-Hackathon-2015: Workshop: Doing Research on Wikimedia things as a volunteer - tools and communities - https://phabricator.wikimedia.org/T91062#1474280 (10Aklapper) What is the status of this task, now that Wikimania 2015 is over? Did this training session take place? If yes: P... [10:41:57] 6Labs, 10Tool-Labs: Mails from tools are being marked as 'spam' by gmail - https://phabricator.wikimedia.org/T104871#1474524 (10Nemo_bis) [10:42:09] 6Labs, 10Tool-Labs, 7Mail: Mails from tools are being marked as 'spam' by gmail - https://phabricator.wikimedia.org/T104871#1430526 (10Nemo_bis) [10:51:08] How long more is the precise distro version supported on tools? [11:50:16] hello, for maps-warper instance, I added labs::lvm::srv role and it may have added 36G space ( /dev/vdb) as /mnt rather than /srv. Is that okay? [11:59:56] 6Labs: Map Warper - No space left on device - https://phabricator.wikimedia.org/T102414#1474652 (10Chippyy) So the ultimate reason for this was that there was no space left and was complicated by the NFS situation with this instance's project coming back online recently. I've freed up a few more megabytes but i... [13:14:54] Coren: would you be able to look at the mailrelay patches in gerrit sometime today or tomorrow? [13:15:40] https://gerrit.wikimedia.org/r/#/c/222362/ and https://gerrit.wikimedia.org/r/#/c/222358/ [13:16:32] and https://phabricator.wikimedia.org/T104733 [13:47:57] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1408 is CRITICAL 40.00% of data above the critical threshold [0.0] [14:15:39] 6Labs, 6Multimedia, 6operations, 10wikitech.wikimedia.org, and 2 others: Some wikitech.wikimedia.org thumbnails broken (404) - https://phabricator.wikimedia.org/T93041#1475035 (10Aklapper) 5Open>3Resolved With the given links in this task and its duplicates I cannot find any broken thumbnails anymore.... [14:27:56] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1408 is OK Less than 1.00% above the threshold [0.0] [15:05:40] 6Labs, 10Tool-Labs, 7I18n: Internationalize Tool Labs' homepage - https://phabricator.wikimedia.org/T105590#1475197 (10scfc) p:5Triage>3Lowest [15:06:44] 6Labs, 10Tool-Labs: exec hosts have apache2 running - https://phabricator.wikimedia.org/T105059#1475205 (10scfc) p:5Triage>3Low [15:11:28] 6Labs, 10Tool-Labs, 7Mail: Mails from tools are being marked as 'spam' by gmail - https://phabricator.wikimedia.org/T104871#1475228 (10scfc) Are there other Gmail users whose mails from Tools get sorted into the spam folder? [15:19:34] 6Labs, 10Tool-Labs: Something went wrong with redis 2015-06-04 to 2015-06-05 - https://phabricator.wikimedia.org/T101514#1475246 (10scfc) >>! In T101514#1422408, @yuvipanda wrote: > Actionable: Note what actually happened, and then close ticket. I still remember, I'll do nexet week? And? :-) [15:21:10] 6Labs, 10Tool-Labs, 7Database: mixnmatch_p on wikidatawiki.labsdb slow - https://phabricator.wikimedia.org/T104833#1475267 (10scfc) [15:21:11] 6Labs, 10Labs-Infrastructure: MySQL spectacularly slow on Labs - https://phabricator.wikimedia.org/T106372#1475268 (10scfc) [15:21:45] 6Labs, 10Tool-Labs, 5Patch-For-Review: new labs host sends out "mpt raid status change" emails - https://phabricator.wikimedia.org/T104779#1475271 (10scfc) (For this to be fixed, after merging the patch new images need to be created and deployed.) [15:23:26] 6Labs, 10Tool-Labs: Decommission Tools-exec-wmt.tools.eqiad.wmflabs - https://phabricator.wikimedia.org/T104919#1475282 (10scfc) a:3scfc [15:23:31] hello everybody [15:24:29] mix-n-match is not working for me today ... is it down, or is the server just too busy? [15:25:47] 6Labs, 6Multimedia, 6operations, 10wikitech.wikimedia.org, and 2 others: Some wikitech.wikimedia.org thumbnails broken (404) - https://phabricator.wikimedia.org/T93041#1475285 (10Krenair) 5Resolved>3Open based on Special:ListFiles a bunch of SVGs and PDFs are still broken [15:26:22] 6Labs, 10Tool-Labs, 15User-Bd808-Test: Create template PHP application for use on Tool Labs based on Slim, Twig and Wikimedia libraries - https://phabricator.wikimedia.org/T90092#1475287 (10bd808) [15:27:11] 6Labs, 10Tool-Labs, 15User-Bd808-Test: Create template PHP application for use on Tool Labs based on Slim, Twig and Wikimedia libraries - https://phabricator.wikimedia.org/T90092#1052395 (10bd808) I didn't get to this one during #Wikimania-Hackathon-2015. I'm going to keep it around though because it will ha... [15:33:28] 6Labs, 10Tool-Labs: Decommission Tools-exec-wmt.tools.eqiad.wmflabs - https://phabricator.wikimedia.org/T104919#1475314 (10scfc) ``` scfc@tools-bastion-01:~$ qconf -dq wmt scfc@tools-bastion-01.eqiad.wmflabs removed "wmt" from cluster queue list scfc@tools-bastion-01:~$ ``` [15:34:35] 6Labs, 10Tool-Labs: Decommission Tools-exec-wmt.tools.eqiad.wmflabs - https://phabricator.wikimedia.org/T104919#1475327 (10scfc) Deleted `tools-exec-wmt` instance. [15:35:01] PROBLEM - Host tools-exec-wmt is DOWN: CRITICAL - Host Unreachable (10.68.16.41) [15:35:07] 6Labs, 10MediaWiki-Configuration, 10wikitech.wikimedia.org: Update wikitech config to allow local logging via Monolog - https://phabricator.wikimedia.org/T106697#1475329 (10bd808) 3NEW [15:40:44] 6Labs, 10Tool-Labs: can't login to tools-shadow-01 - https://phabricator.wikimedia.org/T104781#1475375 (10scfc) This is still happening for me: ``` [tim@passepartout ~]$ ssh tools-shadow-01.eqiad.wmflabs The authenticity of host 'tools-shadow-01.eqiad.wmflabs ()' can't be establis... [15:54:27] YuviPanda: I don't think https://gerrit.wikimedia.org/r/#/c/222468/ is actually working on toollabs :/ [15:56:46] YuviPanda: oh, was actually reverted? [17:01:27] [13intuition] 15siebrand pushed 1 new commit to 06master: 02https://github.com/Krinkle/intuition/commit/572cf42512e96564c9192d2d4887ba88886ceed9 [17:01:28] 13intuition/06master 14572cf42 15Siebrand Mazeland: Localisation updates from https://translatewiki.net. [17:02:42] 6Labs, 10Tool-Labs: "Fatal Error: Database query failed" on "related changes" tool - https://phabricator.wikimedia.org/T105953#1475759 (10scfc) [17:02:43] 10Tool-Labs-tools-Erwin's-tools: "Fatal Error: Database query failed" for Related Changes tool - MySQL down? - https://phabricator.wikimedia.org/T104688#1475760 (10scfc) [17:22:47] 6Labs, 10Tool-Labs: Investigate why Joe is default editor on toollabs - https://phabricator.wikimedia.org/T100526#1475836 (10scfc) 5Open>3declined a:3scfc On `tools-web-static-02` (as an example of a recently created instance) it isn't: ``` scfc@tools-web-static-02:~$ sudo update-alternatives --config e... [18:01:40] 6Labs, 10Labs-Infrastructure: MySQL spectacularly slow on Labs - https://phabricator.wikimedia.org/T106372#1476030 (10Nemo_bis) > and I have to say, you are the top user by BUSY_TIME And I have to say, THANKS Magnus for building tools that users love so much! [18:17:49] 6Labs, 10Labs-Infrastructure: MySQL spectacularly slow on Labs - https://phabricator.wikimedia.org/T106372#1476062 (10Ricordisamoa) >>! In T106372#1466503, @jcrespo wrote: > and I have to say, you are the top user by BUSY_TIME Said about Magnus, this really sounds like a play on words :) [18:23:20] 6Labs, 10Labs-Infrastructure: MySQL spectacularly slow on Labs - https://phabricator.wikimedia.org/T106372#1476084 (10jcrespo) @Magnus, we know labs has it issues (and we are working to improve the service with more servers and better reliability), but: * Did the slowness went away for this specific issue? *... [18:23:33] 10Tool-Labs-tools-Other, 7Epic: Convert all Labs tools to use cdnjs for static libraries - https://phabricator.wikimedia.org/T103934#1476086 (10Ricordisamoa) >>! In T103934#1462369, @Sitic wrote: > https://github.com/simonwhitaker/github-fork-ribbon-css is in cndjs and is a recreation of the Fork me on GitHub... [18:32:51] (03PS1) 10Sitic: Add buttons to edit, revert or unwatch page [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/226589 (https://phabricator.wikimedia.org/T104572) [18:54:55] 6Labs, 10pywikibot-core: pywikipedia.org down? - https://phabricator.wikimedia.org/T106311#1476211 (10Legoktm) @siebrand, any ideas? @valhallasw said you/WMNL registered the domain. [18:57:47] 6Labs: Support bare-metal server allocation in labs - https://phabricator.wikimedia.org/T95185#1476221 (10yuvipanda) [19:00:43] 6Labs: Support bare-metal server allocation in labs - https://phabricator.wikimedia.org/T95185#1476228 (10yuvipanda) I'm going to test this with real hardware, and we can come up with an allocation process for it later. Things to do to make it know it is labs: # Set 'realm' to labs # Set hostname to something.... [19:02:39] 6Labs, 10pywikibot-core: pywikipedia.org down? - https://phabricator.wikimedia.org/T106311#1476233 (10valhallasw) ``` valhallasw@maeglin:operations-puppet$ nslookup... [19:10:45] 6Labs, 10pywikibot-core: pywikipedia.org down? - https://phabricator.wikimedia.org/T106311#1476249 (10valhallasw) Was removed with https://github.com/wikimedia/operations-dns/commit/a782e3adcb9cdd046b9709870ad4f49dcc992890 So the cname is wrong; the question is: what should it be replaced with...? [19:10:54] 6Labs, 10Labs-Infrastructure, 7Composer, 7Upstream: Composer activity from Labs hosts can be rate limited by GitHub - https://phabricator.wikimedia.org/T106452#1476251 (10yuvipanda) @bd808 208.80.155.128/25 is the labs CIDR [19:57:30] 6Labs, 10Labs-Infrastructure, 10hardware-requests, 6operations: New server: labdns1001 - https://phabricator.wikimedia.org/T106147#1476390 (10RobH) a:5RobH>3Andrew [20:18:14] 6Labs, 10Labs-Infrastructure: MySQL spectacularly slow on Labs - https://phabricator.wikimedia.org/T106372#1476462 (10Magnus) @jcrespo : - The issue went away at the time, but it's back at the moment. The tool (mix-n-match) is not working because of it. Since this issue has appeared many times in the past,... [20:20:33] 6Labs, 10Labs-Infrastructure: MySQL spectacularly slow on Labs - https://phabricator.wikimedia.org/T106372#1476472 (10Magnus) Update: Tool now seems to work ATM. Fingers crossed. [20:41:04] 6Labs, 10Tool-Labs, 10Wikimedia-Git-or-Gerrit: git clone operations/mediawiki-config on tool labs fail: recursion detected in die_errno handler - https://phabricator.wikimedia.org/T106393#1476552 (10scfc) Works for me in #Tool-Labs: ``` scfc@tools-bastion-01:~$ git clone https://git.wikimedia.org/git/operat... [21:37:14] 6Labs, 10Tool-Labs: Permission issues and/or failure to load Ruby environment on trusty - https://phabricator.wikimedia.org/T106170#1476840 (10scfc) (I have removed the `o+w` right for `~tools.musikbot/exec.sh` and (recursively) `~tools.musikbot/MusikBot`. I couldn't find any files //not// belonging to that u... [22:07:24] If I make a file that required heavy processing on an instance, but I want my tool to be able to see it, where should I put it, don't they share a filesystem? so i don't need to copy it over? [22:07:54] howerevr is chmod 755'd the dir on my instance, and my tool says that permission is denied [22:09:57] 6Labs, 10Labs-Infrastructure, 7Composer, 7Upstream: Composer activity from Labs hosts can be rate limited by GitHub - https://phabricator.wikimedia.org/T106452#1477005 (10bd808) I sent this to GitHub: > Before I get to whining and asking for favors, thanks for making such a great platform for the FLOSS co... [22:12:40] 6Labs, 10Labs-Infrastructure, 7Composer, 7Upstream: Composer activity from Labs hosts can be rate limited by GitHub - https://phabricator.wikimedia.org/T106452#1477022 (10bd808) a:3bd808 [22:13:28] 6Labs, 10MediaWiki-Configuration, 10wikitech.wikimedia.org: Update wikitech config to allow local logging via Monolog - https://phabricator.wikimedia.org/T106697#1477024 (10bd808) [22:22:26] 6Labs, 10Tool-Labs: 'webservice not available' message no longer shown - https://phabricator.wikimedia.org/T104870#1477051 (10scfc) To clarify: You mean that https://tools.wmflabs.org/fdafdsaf should say "unknown tool" or something to that effect? Because https://tools.wmflabs.org/abibot/ //does// say "No web... [22:24:37] 6Labs, 6Multimedia, 6operations, 10wikitech.wikimedia.org, and 2 others: Some wikitech.wikimedia.org thumbnails broken (404) - https://phabricator.wikimedia.org/T93041#1477077 (10Bawolff) PDFs I would guess is missing the pdfinfo command line tool (popler-utils package i think). [This is a guess though] S... [22:29:12] 6Labs, 6Multimedia, 6operations, 10wikitech.wikimedia.org, and 2 others: Some wikitech.wikimedia.org thumbnails broken (404) - https://phabricator.wikimedia.org/T93041#1477112 (10Bawolff) >>! In T93041#1371054, @Krenair wrote: > Ugh, why another wikitech-specific hack? What about @bawolff's comment? There... [22:29:18] 6Labs, 6Multimedia, 6operations, 10wikitech.wikimedia.org, and 2 others: Some wikitech.wikimedia.org thumbnails broken (404) - https://phabricator.wikimedia.org/T93041#1477113 (10Krenair) >>! In T93041#1477077, @Bawolff wrote: > PDFs I would guess is missing the pdfinfo command line tool (popler-utils pack... [22:36:19] 6Labs, 6Multimedia, 6operations, 10wikitech.wikimedia.org, and 2 others: Some wikitech.wikimedia.org thumbnails broken (404) - https://phabricator.wikimedia.org/T93041#1477185 (10Krenair) >>! In T93041#1477077, @Bawolff wrote: > I believe that --no-external-files is a flag that comes from a WMF specific pa... [22:38:50] 6Labs, 10Labs-Infrastructure: MySQL spectacularly slow on Labs - https://phabricator.wikimedia.org/T106372#1477201 (10Magnus) 5Open>3declined a:3Magnus - Spent a few hours moving the database to the tools-db server, and changed a bucket of files to use the new database instead. - Rewrote about a dozen in... [22:50:31] 6Labs, 6Multimedia, 6operations, 10wikitech.wikimedia.org, and 2 others: Some wikitech.wikimedia.org thumbnails broken (404) - https://phabricator.wikimedia.org/T93041#1477263 (10Krenair) See also T104147, T80392 [23:46:47] 6Labs, 10Tool-Labs: Permission issues and/or failure to load Ruby environment on trusty - https://phabricator.wikimedia.org/T106170#1477419 (10Sitic) I suspect this is a stale NFS file handle issue, I had a similar problem today. The problem MusikAnimal has/had is intermitting, getting `Permission denied` som... [23:47:49] (03CR) 10Sitic: [C: 032 V: 032] Add buttons to edit, revert or unwatch page [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/226589 (https://phabricator.wikimedia.org/T104572) (owner: 10Sitic) [23:47:58] How long more is the precise distro version supported on tools? [23:49:35] svetlana: there's no end date for it. it'll probably be made non-default at some point, but probably will be supported until it gets EOL'd? another 2y? [23:49:53] Oh. Good. Thank you. [23:50:13] svetlana: yw. basically we'll drop support for it whenever Ubuntu drops support for it.