[00:10:16] 06Labs, 10Tool-Labs, 06Collaboration-Team-Triage, 06Community-Tech-Tool-Labs, and 5 others: Enable Flow on wikitech (labswiki and labtestwiki), then turn on for Tool talk namespace - https://phabricator.wikimedia.org/T127792#2621619 (10Dereckson) [00:11:02] 06Labs, 10Tool-Labs, 06Collaboration-Team-Triage, 06Community-Tech-Tool-Labs, and 5 others: Enable Flow on wikitech (labswiki and labtestwiki), then turn on for Tool talk namespace - https://phabricator.wikimedia.org/T127792#2054159 (10Dereckson) [00:12:22] 06Labs, 10Tool-Labs, 06Collaboration-Team-Triage, 06Community-Tech-Tool-Labs, and 5 others: Enable Flow on wikitech (labswiki and labtestwiki), then turn on for Tool talk namespace - https://phabricator.wikimedia.org/T127792#2621627 (10Dereckson) [01:42:05] 10Striker, 10Phabricator, 10Security-Reviews, 13Patch-For-Review: Unable to mirror repository from git.legoktm.com into diffusion - https://phabricator.wikimedia.org/T143969#2621891 (10mmodell) Would this fall under security-reviews? [02:06:41] 10Striker: Allow easy replication of existing github/bitbucket repos - https://phabricator.wikimedia.org/T143971#2584987 (10Legoktm) >>! In T143971#2594053, @Luke081515 wrote: > Alternativly, if a user thinks, that he can't do that manully, I can do that for them if wished, just assign then a task to me ;) I be... [03:03:46] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Clean up leaked designate entries - https://phabricator.wikimedia.org/T120797#2622061 (10AlexMonk-WMF) a:05AlexMonk-WMF>03None Waiting for ops [03:04:14] 06Labs, 10Horizon, 13Patch-For-Review: Switch dynamicproxy to point back to IP rather than domain names - https://phabricator.wikimedia.org/T133554#2622067 (10AlexMonk-WMF) @yuvipanda [05:55:15] 06Labs, 10Labs-Infrastructure, 10DBA, 07Upstream: mysqld process hang in db1069 - S2 mysql instance - https://phabricator.wikimedia.org/T145077#2622297 (10Marostegui) I will open a bug report to tokudb today and paste here the link to it [06:44:36] PROBLEM - Puppet staleness on tools-exec-1410 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [43200.0] [08:21:58] !log tools.wikiloves Edited templates/eventmain.html on the server to avoid syntax error. [08:22:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikiloves/SAL, Master [08:23:31] !log tools.wikiloves Add .description at the root in order to display a short description in the tools list [08:23:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikiloves/SAL, Master [09:04:05] PROBLEM - SSH on tools-webgrid-lighttpd-1210 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:05:02] 06Labs, 10Continuous-Integration-Infrastructure: Request increased quota for contintcloud labs project - https://phabricator.wikimedia.org/T142877#2622524 (10hashar) I am closing this task. The quota used to be 20 until July 4th when it got lowered down to 10 in an emergency due to wmflabs being full. We ge... [09:05:17] 06Labs, 10Labs-Infrastructure, 05Continuous-Integration-Scaling, 13Patch-For-Review: Bump quota of Nodepool instances (contintcloud tenant) - https://phabricator.wikimedia.org/T133911#2622528 (10hashar) [09:05:19] 06Labs, 10Continuous-Integration-Infrastructure: Request increased quota for contintcloud labs project - https://phabricator.wikimedia.org/T142877#2622530 (10hashar) [09:09:17] 06Labs, 06Operations: Puppet broken on labcontrol1002 - https://phabricator.wikimedia.org/T145185#2622533 (10Volans) [10:18:21] 06Labs: cronspam from labscontrol1001, labstore1001, labnet1002.eqiad.wmnet, labsdb1003.eqiad.wmnet - https://phabricator.wikimedia.org/T132422#2622668 (10elukey) Still getting regular emails about gzip: stdin: file size changed while zipping, it shouldn't be anything related to the logrotate files in operations... [10:21:07] 06Labs: cronspam from labscontrol1001, labstore1001, labnet1002.eqiad.wmnet, labsdb1003.eqiad.wmnet - https://phabricator.wikimedia.org/T132422#2198026 (10MoritzMuehlenhoff) Maybe these are coming from unpuppetised base services installed by Debian/Ubuntu? [10:23:09] (03CR) 10Lokal Profil: [C: 032] Refactor database configuration handling [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/303428 (owner: 10Jean-Frédéric) [10:23:51] (03Merged) 10jenkins-bot: Refactor database configuration handling [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/303428 (owner: 10Jean-Frédéric) [10:43:56] RECOVERY - SSH on tools-webgrid-lighttpd-1210 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [10:49:59] PROBLEM - SSH on tools-webgrid-lighttpd-1210 is CRITICAL: Server answer [11:09:56] RECOVERY - SSH on tools-webgrid-lighttpd-1210 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [11:15:55] PROBLEM - SSH on tools-webgrid-lighttpd-1210 is CRITICAL: Server answer [11:16:04] (03CR) 10Jean-Frédéric: "Thanks for merging, and for the reviews! I must say I’m quite happy with that patch − makes things waaaay cleaner ^__^" (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/303428 (owner: 10Jean-Frédéric) [11:19:41] (03CR) 10Jean-Frédéric: [C: 032] Setup local development environment for ErfgoedBot [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/303498 (owner: 10Jean-Frédéric) [11:20:22] (03Merged) 10jenkins-bot: Setup local development environment for ErfgoedBot [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/303498 (owner: 10Jean-Frédéric) [11:45:24] (03CR) 10Jean-Frédéric: "recheck" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309324 (owner: 10Jean-Frédéric) [12:31:13] 06Labs, 10Labs-Infrastructure, 10DBA, 07Upstream: mysqld process hang in db1069 - S2 mysql instance - https://phabricator.wikimedia.org/T145077#2622875 (10Marostegui) I have opened the bug report: https://bugs.launchpad.net/percona-server/+bug/1621852 [12:31:29] 06Labs, 10Labs-Infrastructure, 10DBA, 07Upstream: mysqld process hang in db1069 - S2 mysql instance - https://phabricator.wikimedia.org/T145077#2622876 (10Marostegui) a:03Marostegui [12:35:56] RECOVERY - SSH on tools-webgrid-lighttpd-1210 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [12:39:02] hi, there is a Tool Labs job (310763) in status "Task / Deleting" from 30 August (there are other jobs in this condition). What should I do for deleting it? Qdel doesn't work. [12:41:55] PROBLEM - SSH on tools-webgrid-lighttpd-1210 is CRITICAL: Server answer [12:45:34] indeed if I use: "$ qdel 310763" I get: "job 310763 is already in deletion" (from August 30) [12:47:31] rotpunkt: I can try a force delete [12:47:37] done [12:49:28] ok thanks [12:49:51] Is it a command also for normal users? Can I use it next time? [12:50:42] "-f"? [12:55:33] rotpunkt: yep '-f' and I thikn it's restricted to Tools admins so you'd have to ask here again in the hopefully rare case [13:09:34] ok thanks again, see you! [13:19:16] (03CR) 10Lokal Profil: [C: 04-1] "-1 mainly because this patch only works together with the next one (where my two comments have been fixed)." (032 comments) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309324 (owner: 10Jean-Frédéric) [13:37:23] (03CR) 10Lokal Profil: [C: 04-1] "Yay for implementing this!" (034 comments) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309325 (https://phabricator.wikimedia.org/T114166) (owner: 10Jean-Frédéric) [15:10:22] 06Labs: Request creation of labs-vpn labs project - https://phabricator.wikimedia.org/T143939#2623157 (10bd808) [15:28:58] hi, vem.maps-team.eqiad.wmflabs is stuck in reboot limbo - needs powercycling :( [15:31:46] aha! its alive :) [15:48:37] (03PS3) 10Lokal Profil: Replace TestFillTableMonumentsBase by CustomAssertions [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/302887 [15:49:39] (03CR) 10Lokal Profil: "I had to rewrite the assert a bit to handle differently formatted "msg" parameters." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/302887 (owner: 10Lokal Profil) [17:02:25] 06Labs: Request creation of labs-vpn labs project - https://phabricator.wikimedia.org/T143939#2623510 (10Multichill) Some considerations if you want to go down the VPN road: * VPN should be at least as secure as our current SSH setup or better * We're talking about user vpn's here, not site-to-site vpn's * VPN's... [17:08:28] 06Labs: Request creation of labs-vpn labs project - https://phabricator.wikimedia.org/T143939#2623514 (10AlexMonk-WMF) >>! In T143939#2623510, @Multichill wrote: > * I would integrate it with the central LDAP to keep users in one place. Having multiple LDAP instances is a PITA and a security incident waiting to... [17:14:09] 06Labs, 06Operations: Enable root passwords on Labs VMs - https://phabricator.wikimedia.org/T142216#2623524 (10Andrew) [17:14:11] 06Labs, 13Patch-For-Review: Don't set instance root passwords if using a local puppetmaster - https://phabricator.wikimedia.org/T142531#2623523 (10Andrew) 05Open>03Resolved [17:32:06] * Alphos dances around [17:32:15] wrcr works as intended ! \o/ [17:32:27] cron job went just fine ^_^ [17:34:27] and reporting on the reporting went fine as well, yay \o/ [17:35:00] that means i should probably start working on a visualization interface of sorts ^^' [17:37:08] Alphos: congratulations :) [17:37:42] thanks :) [17:39:04] by the way, if anyone wants to remove links from wikidata items to wikipedia pages that are redirects to other pages, https://tools.wmflabs.org/wikidata-redirects-conflicts-reports/reports/2016-36/ have fun ! ^_^ [17:43:28] 06Labs, 10Wikimedia-Extension-setup, 10wikitech.wikimedia.org, 07I18n, and 2 others: Install Translate extension on wikitech - https://phabricator.wikimedia.org/T100313#2623691 (10Jdforrester-WMF) >>! In T100313#2618318, @Dereckson wrote: > @Jdforrester-WMF What are your fear and what would you like to avo... [18:00:15] is it me or are labs ssh connections a wee bit on the slow side ? [18:08:48] guys, picking up something really weird on tools-bastion-03. there's a grep call munching >5G in both VIRT and RES [18:09:07] and tools-bastion-03 is really slow :/ [18:10:54] !log tools killed massive grep running as root [18:10:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [18:12:23] oh yeah, forgot to mention it was running as root ^^' [18:12:33] part of the weirdness [18:13:49] yuvipanda : killing it doesn't seem to have done it. stake to the heart or silver bullet to the head perhaps ? [18:13:55] (or -9 :p ) [18:16:44] (still appears in htop, and tools-bastion-03 is no less sluggish) [18:39:30] maxsem@vem2:/srv/mediawiki-vagrant$ vagrant provision [18:39:30] No usable default provider could be found for your system. [18:40:11] Alphos, sorry, just saw this [18:43:32] Krenair no problem, mysql-client seems to work about fine for what i'm doing [18:44:33] Alphos, it should be really gone now [18:45:14] thanks [19:18:53] may I poke about this new project request https://phabricator.wikimedia.org/T144388 ? Especially if the answer is negative, I should be start working on alternative solution soon [19:22:43] 10Labs-project-Librarybase, 10Reports-bot, 10The-Wikipedia-Library, 10WikiCite: Create recommendations for databases/journals/websites, by WikiProject for WikiProject X - https://phabricator.wikimedia.org/T111066#2624070 (10Quiddity) [19:27:54] !log tools reboot tools-exec-1218 and 1219 [19:27:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [19:28:03] 06Labs, 10Tool-Labs: Gridengine nodes tools-exec-1218 and 1219 seem unreachable - https://phabricator.wikimedia.org/T144789#2624098 (10yuvipanda) I'm just going to reboot them and let them be now. [19:32:04] RECOVERY - SSH on tools-exec-1219 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [19:32:22] 06Labs, 10wikitech.wikimedia.org, 13Patch-For-Review: mwscriptwikiset broken when using all.dblist on terbium - https://phabricator.wikimedia.org/T132383#2624119 (10demon) 05Open>03Resolved a:03demon [19:32:31] 06Labs, 10wikitech.wikimedia.org, 13Patch-For-Review: Special page reports not updating on Wikitech - https://phabricator.wikimedia.org/T136926#2624133 (10demon) 05Open>03Resolved a:03demon [19:35:00] PROBLEM - Puppet run on tools-exec-1219 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [19:37:19] PROBLEM - Puppet run on tools-docker-builder-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:37:48] 06Labs, 10Tool-Labs: Gridengine nodes tools-exec-1218 and 1219 seem unreachable - https://phabricator.wikimedia.org/T144789#2624143 (10yuvipanda) 05Open>03Resolved a:03yuvipanda I've repooled them. [19:38:07] PROBLEM - Puppet staleness on tools-exec-1219 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [43200.0] [19:43:09] RECOVERY - Puppet staleness on tools-exec-1219 is OK: OK: Less than 1.00% above the threshold [3600.0] [19:49:08] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Jfhutson was created, changed by Jfhutson link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Jfhutson edit summary: Created page with "{{Tools Access Request |Justification=Fixing a broken tool used by Wikipedia |Completed=false |User Name=Jfhutson }}" [19:49:58] RECOVERY - Puppet run on tools-exec-1219 is OK: OK: Less than 1.00% above the threshold [0.0] [19:51:51] andrewbogott: Just discovered I had like 6 floating IPs allocated to a project...when I had a quota of 1. I gave you 5 back :p [20:10:21] ostriches: thanks! [20:26:18] speaking of floating ips [20:26:33] chasemp: I got a lame patch to get Nodepool to stop querying the list of floating ips (which we dont need) [20:26:47] https://gerrit.wikimedia.org/r/#/c/309464/ and I have build the new .deb package. Might want to deploy that sometime next week [20:27:32] though I am not sure what is the best channel to ask for a slice of you guys time :) [20:28:45] where is the patch itself? [20:37:45] https://gerrit.wikimedia.org/r/#/c/309406/ [20:37:50] https://gerrit.wikimedia.org/r/#/c/309435/ [20:39:50] MaxSem: I think that error usually means that your shell session is missing the alias vagrant=/usr/local/bin/mwvagrant. There is an /etc/profile.d script provisioned by Puppet that sets that up [20:42:08] bd808, ah. relogined, works - thanks! [20:43:22] Change on 12www.mediawiki.org a page OAuth/For Developers was modified, changed by Jdlrobson link https://www.mediawiki.org/w/index.php?diff=2234824 edit summary: Explain how to use with Node.js [20:47:20] chasemp: sorry I went crazy yesterday. Short circuit is https://gerrit.wikimedia.org/r/#/c/309406/1/nodepool/provider_manager.py [21:00:46] MaxSem: yw. sorry the error message is so unhelpful there