[00:02:51] 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: rcstream not working for wikitech wiki - https://phabricator.wikimedia.org/T136245#2332584 (10Krenair) 05Open>03Resolved That fixed it, thanks @dzahn! [00:03:16] 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: rcstream not working for wikitech wiki - https://phabricator.wikimedia.org/T136245#2332586 (10Dzahn) @aude ^ [00:03:57] 06Labs, 10Labs-Infrastructure, 06Operations: rcstream not working for wikitech wiki - https://phabricator.wikimedia.org/T136245#2332592 (10Dzahn) [02:48:41] 06Labs, 10MediaWiki-extensions-OATHAuth, 06Security-Team, 10wikitech.wikimedia.org, 13Patch-For-Review: wikitech 2fa provisioning form does so without confirmation - https://phabricator.wikimedia.org/T130892#2332838 (10dpatrick) @jcrespo, where you able to recover this data? @csteipp, if jcrespo was unab... [05:20:50] RECOVERY - Puppet run on tools-exec-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [05:58:58] 06Labs, 10MediaWiki-extensions-OATHAuth, 06Security-Team, 10wikitech.wikimedia.org, 13Patch-For-Review: wikitech 2fa provisioning form does so without confirmation - https://phabricator.wikimedia.org/T130892#2332916 (10jcrespo) @dpatrick No, I wasn't. Backups are now working, but they failed for the rang... [07:44:40] RECOVERY - Puppet run on tools-webgrid-lighttpd-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [08:43:43] (03PS1) 10Jean-Frédéric: Add ReadMe file [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/291196 [08:43:45] (03PS1) 10Jean-Frédéric: Move/Merge initialization bits from api.php to common.php [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/291197 [08:43:47] (03PS1) 10Jean-Frédéric: Add local dev environment with docker-compose [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/291198 (https://phabricator.wikimedia.org/T136351) [09:08:05] (03PS2) 10Jean-Frédéric: Add local dev environment with docker-compose [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/291198 (https://phabricator.wikimedia.org/T136351) [09:39:08] 06Labs, 10Tool-Labs: Fwd: Alert: puppet failed on toolsbeta hosts - https://phabricator.wikimedia.org/T136317#2333139 (10valhallasw) Warning for toolsbeta-webgrid-lighttpd-1201 tonight, which is another high load issue. Rebooted. toolsbeta-mail looks OK in terms of load & puppet runs, so I'm assuming the rest... [09:47:52] (03CR) 10Lokal Profil: [C: 032] Add ReadMe file [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/291196 (owner: 10Jean-Frédéric) [09:48:42] (03Merged) 10jenkins-bot: Add ReadMe file [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/291196 (owner: 10Jean-Frédéric) [09:51:47] (03CR) 10Lokal Profil: [C: 032] Move/Merge initialization bits from api.php to common.php [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/291197 (owner: 10Jean-Frédéric) [09:52:35] (03Merged) 10jenkins-bot: Move/Merge initialization bits from api.php to common.php [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/291197 (owner: 10Jean-Frédéric) [09:53:30] (03CR) 10Lokal Profil: "This should be considered a stop-gap measure to ensure no one breaks the config." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/290557 (owner: 10Lokal Profil) [09:54:46] (03PS3) 10Jean-Frédéric: Add local dev environment with docker-compose [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/291198 (https://phabricator.wikimedia.org/T136351) [09:58:10] (03CR) 10Jean-Frédéric: Add test to validate entries in monuments_config (032 comments) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/290557 (owner: 10Lokal Profil) [10:10:06] 06Labs, 10Tool-Labs: Prevent overly-large log files - https://phabricator.wikimedia.org/T122508#2333201 (10valhallasw) > So we went from considering deletion too heavy-handed I don't think we considered deletion too heavy-handed, but we try to do this while considering the needs of users. > to being fine with... [10:10:35] RECOVERY - Puppet run on tools-webgrid-lighttpd-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [11:04:23] 06Labs, 10Tool-Labs: Set up replica from extension1 cluster - https://phabricator.wikimedia.org/T121742#2333264 (10valhallasw) p:05Triage>03Low [11:04:41] 06Labs, 10MediaWiki-extensions-OATHAuth: Move two-factor auth data (TOTP seed) from labswiki database to LDAP - https://phabricator.wikimedia.org/T136350#2333265 (10valhallasw) [11:05:14] 06Labs, 10Tool-Labs: Backup and/or puppetize @toolserver.org mail forwards - https://phabricator.wikimedia.org/T136225#2333267 (10valhallasw) p:05Triage>03Normal [11:05:46] 06Labs, 10Tool-Labs, 13Patch-For-Review: Investigate Tool Labs webservice outage on 2016-05-25 - https://phabricator.wikimedia.org/T136162#2333269 (10valhallasw) [11:05:48] 06Labs, 10Tool-Labs, 13Patch-For-Review: Fix webservice start check from tools webservice - https://phabricator.wikimedia.org/T136347#2333268 (10valhallasw) 05Open>03Resolved [11:05:59] 06Labs, 10Tool-Labs, 10DBA: Allow self-serve database credential and permissions management for Tool Labs projects - https://phabricator.wikimedia.org/T136335#2333270 (10valhallasw) p:05Triage>03Lowest [11:07:35] 06Labs, 10Tool-Labs: Fwd: Alert: puppet failed on toolsbeta hosts - https://phabricator.wikimedia.org/T136317#2333272 (10valhallasw) 05Open>03Resolved p:05Triage>03High [11:07:58] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 06Community-Tech-Tool-Labs: Develop evaluation criteria for comparing PaaS solutions - https://phabricator.wikimedia.org/T136265#2333274 (10valhallasw) p:05Triage>03Low [11:08:14] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Epic: Evalute Kubernetes based workflow replacement options for SGE - https://phabricator.wikimedia.org/T136264#2333276 (10valhallasw) p:05Triage>03Low [11:09:07] 06Labs, 10Tool-Labs: Rounding and missing units in VMEM values on http://tools.wmflabs.org/?status create misleading values - https://phabricator.wikimedia.org/T119680#2333284 (10valhallasw) p:05Lowest>03High [11:09:08] 06Labs, 10Tool-Labs, 07Easy: Log CommitLimit to graphite - https://phabricator.wikimedia.org/T118465#2333287 (10valhallasw) p:05Lowest>03High [11:09:14] 06Labs, 10Tool-Labs: Make sure all packages are automatically upgraded - https://phabricator.wikimedia.org/T118939#2333294 (10valhallasw) p:05Low>03High [11:09:16] 06Labs, 10Tool-Labs: add user filter to status page - https://phabricator.wikimedia.org/T75384#2333297 (10valhallasw) p:05Lowest>03High [11:09:17] wtf?! [11:10:42] 06Labs, 10Tool-Labs: Rounding and missing units in VMEM values on http://tools.wmflabs.org/?status create misleading values - https://phabricator.wikimedia.org/T119680#1832833 (10valhallasw) p:05High>03Lowest [11:10:46] 06Labs, 10Tool-Labs, 07Easy: Log CommitLimit to graphite - https://phabricator.wikimedia.org/T118465#2333314 (10valhallasw) p:05High>03Lowest [11:10:50] 06Labs, 10Tool-Labs: Make sure all packages are automatically upgraded - https://phabricator.wikimedia.org/T118939#1813269 (10valhallasw) p:05High>03Low [11:11:15] 06Labs, 10Tool-Labs: add user filter to status page - https://phabricator.wikimedia.org/T75384#2333316 (10valhallasw) p:05High>03Lowest [11:12:08] 06Labs, 10Tool-Labs: Add all service group users to a project-servicegroups-$project by default - https://phabricator.wikimedia.org/T116815#2333320 (10valhallasw) p:05Low>03Lowest [11:12:15] Yes, this is a 'feature' [11:12:16] Great. [11:13:53] 06Labs, 10Tool-Labs: Add all service group users to a project-servicegroups-$project by default - https://phabricator.wikimedia.org/T116815#1759101 (10valhallasw) p:05Lowest>03Low Sorry about this - see {T136400} for details. [11:21:05] 06Labs, 10Tool-Labs: Prevent overly-large log files - https://phabricator.wikimedia.org/T122508#2333349 (10Nemo_bis) I'm not sure why it would be a significant engineering effort, it's just a matter of issuing one `find` command (with ionice?). But sure, disk quotas are a common solution. [11:34:12] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Tracking: Missing Toolserver features in Tools (tracking) - https://phabricator.wikimedia.org/T60791#2333367 (10valhallasw) [11:34:14] 06Labs, 10Tool-Labs, 10Diffusion, 10Internet-Archive: Copy contents of https://svn.toolserver.org/ to Wikimedia Diffusion - https://phabricator.wikimedia.org/T60801#2333364 (10valhallasw) 05Open>03Resolved a:03mmodell From the list of repositories, all of them seem to have been imported. If something... [11:38:37] 06Labs, 10Tool-Labs: Prevent overly-large log files - https://phabricator.wikimedia.org/T122508#1906120 (10Ladsgroup) syslog and daemon.log works this way: - syslog <-- log of today - syslog.1 <-- log of yesterday - syslog.2.gz (zipped) <-- log of the day before And so on. It would be great to have su... [11:40:17] 06Labs, 10Tool-Labs: Set up replica from extension1 cluster - https://phabricator.wikimedia.org/T121742#2333373 (10jcrespo) 05Open>03declined I would say to decline for now (that doesn't mean we cannot talk about setting it at some point in the future, but this is now technically not possible - due to spac... [11:41:04] 06Labs, 10Tool-Labs: Prevent overly-large log files - https://phabricator.wikimedia.org/T122508#2333377 (10valhallasw) Unfortunately that's not possible without straining NFS even more. SGE does not allow move-and-write-to-new-file style logrotation, which means we have to use copy-and-truncate, which would do... [12:18:17] 06Labs, 10Tool-Labs, 13Patch-For-Review: Reduce amount of Tools-local packages - https://phabricator.wikimedia.org/T91874#2333443 (10valhallasw) [12:18:40] 06Labs, 10Tool-Labs: Define policy for tools-local packages - https://phabricator.wikimedia.org/T114645#2333440 (10valhallasw) 05Open>03Resolved a:03valhallasw Policy is on-wiki at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Admin. [12:19:23] 06Labs, 10Tool-Labs, 13Patch-For-Review: Reduce amount of Tools-local packages - https://phabricator.wikimedia.org/T91874#1097938 (10valhallasw) [12:20:13] 06Labs, 10Tool-Labs, 13Patch-For-Review: webservicemonitor: AttributeError: 'module' object has no attribute 'utcnow' - https://phabricator.wikimedia.org/T115225#2333448 (10valhallasw) 05Open>03Resolved a:03coren [12:22:59] 06Labs, 10Tool-Labs, 10Mail: tools-mail: check SPF of sender before forwarding email - https://phabricator.wikimedia.org/T120210#2333460 (10valhallasw) [12:23:40] 06Labs, 10Tool-Labs: cron fail on tools-submit - https://phabricator.wikimedia.org/T121305#2333468 (10valhallasw) 05Open>03Resolved [12:24:32] 06Labs, 10Tool-Labs: queues on tools-exec-1402.eqiad.wmflabs and tools-exec-1203.eqiad.wmflabs are disabled - https://phabricator.wikimedia.org/T122125#2333469 (10valhallasw) 05Open>03Resolved [12:25:31] 06Labs, 10Tool-Labs: Queues disabled on tools-exec-1407.eqiad.wmflabs, tools-exec-1216.eqiad.wmflabs, tools-exec-1219.eqiad.wmflabs - https://phabricator.wikimedia.org/T136404#2333482 (10valhallasw) [12:26:12] 06Labs, 10Tool-Labs, 13Patch-For-Review: 70.179.6.X is hammering xtools-articleinfo - https://phabricator.wikimedia.org/T122307#2333495 (10valhallasw) 05Open>03Resolved a:03valhallasw https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Banned [12:27:15] (03PS1) 10Merlijn van Deen: Block xtools-articleinfo from scrapers [labs/toollabs] - 10https://gerrit.wikimedia.org/r/291214 (https://phabricator.wikimedia.org/T122327) [12:27:33] (03CR) 10jenkins-bot: [V: 04-1] Block xtools-articleinfo from scrapers [labs/toollabs] - 10https://gerrit.wikimedia.org/r/291214 (https://phabricator.wikimedia.org/T122327) (owner: 10Merlijn van Deen) [12:27:49] (03PS2) 10Merlijn van Deen: Block xtools-articleinfo from scrapers [labs/toollabs] - 10https://gerrit.wikimedia.org/r/291214 (https://phabricator.wikimedia.org/T122327) [12:35:09] 06Labs, 10Tool-Labs: Labs users should be able to force-delete their own jobs - https://phabricator.wikimedia.org/T127681#2333521 (10valhallasw) 05Open>03Invalid I'd rather not, for several reasons. First, this requires touching SGE config files, which is always risky. Secondly, this can easily cause users... [12:36:13] 06Labs, 10Tool-Labs: Toolsbeta bigbrother failing to start toolhistory - https://phabricator.wikimedia.org/T123197#2333524 (10valhallasw) 05Open>03Invalid [12:36:55] 06Labs, 10Tool-Labs: toolsbeta-webproxy-01 not accessible - https://phabricator.wikimedia.org/T122802#2333525 (10valhallasw) 05Open>03Resolved a:03scfc [12:37:21] 06Labs, 10Tool-Labs: tools-bastion-02 /dev/vda1 (root mount) 100% full - https://phabricator.wikimedia.org/T122713#2333528 (10valhallasw) 05Open>03Resolved [12:37:55] 06Labs, 10Tool-Labs: Apply pretty 'banned' error page to user-agent bans - https://phabricator.wikimedia.org/T122583#2333529 (10valhallasw) p:05Triage>03Lowest [12:38:16] 06Labs, 10Tool-Labs, 13Patch-For-Review: webservice calls get stuck and do not timeout sometimes when called from toolschecker - https://phabricator.wikimedia.org/T123987#2333532 (10valhallasw) 05Open>03Resolved a:03yuvipanda [12:40:12] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Other: Find a method for `templatetiger` to sort large files - https://phabricator.wikimedia.org/T124822#2333535 (10valhallasw) p:05Triage>03Low [12:40:33] 06Labs, 10Tool-Labs, 06Security-Team: consider making individual tools on tool labs have their own X.tools.wmflabs.org subdomain - https://phabricator.wikimedia.org/T125589#2333537 (10valhallasw) p:05Triage>03Lowest [12:40:50] 06Labs, 10Tool-Labs: toolserver.org uses 302 redirects instead of 301 - https://phabricator.wikimedia.org/T123861#2333540 (10valhallasw) 05Open>03Invalid [12:46:31] 06Labs, 10Tool-Labs, 10WLX-Jury: Figure out a way to support java 1.8 on tool labs (For WLX Jury) - https://phabricator.wikimedia.org/T124903#2333546 (10valhallasw) p:05Triage>03Low Unfortunately, there is no solution for this in the very near future. @yuvipanda is working on the kubernetes solutions, an... [12:46:53] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Documentation: Create a "my first PHP webservice" tutorial for Tool Labs - https://phabricator.wikimedia.org/T134493#2333548 (10valhallasw) p:05Low>03Lowest [12:47:02] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Documentation: Create a "my first Python webservice" tutorial for Tool Labs - https://phabricator.wikimedia.org/T134494#2333551 (10valhallasw) p:05Low>03Lowest [12:47:08] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Documentation: Create a "my first Pywikibot bot" tutorial for Tool Labs - https://phabricator.wikimedia.org/T134495#2333554 (10valhallasw) p:05Low>03Lowest [12:47:20] 06Labs, 10Tool-Labs, 10Monitoring, 06Operations: Make icinga-wm report Tools homepage check at #wikimedia-labs, too - https://phabricator.wikimedia.org/T128716#2333557 (10valhallasw) p:05Low>03High [12:48:44] AAAAH [12:48:48] grrr [12:49:10] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Documentation: Create a "my first PHP webservice" tutorial for Tool Labs - https://phabricator.wikimedia.org/T134493#2333561 (10valhallasw) p:05Lowest>03Low [12:49:18] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 06Developer-Relations, 07Documentation: Run a documentation sprint for Labs - https://phabricator.wikimedia.org/T101659#2333565 (10valhallasw) [12:49:21] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Documentation: Create a "my first Python webservice" tutorial for Tool Labs - https://phabricator.wikimedia.org/T134494#2333562 (10valhallasw) 05Open>03Resolved p:05Lowest>03Low a:03valhallasw [12:49:24] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Documentation: Create a "my first Pywikibot bot" tutorial for Tool Labs - https://phabricator.wikimedia.org/T134495#2333566 (10valhallasw) p:05Lowest>03Low [12:49:28] 06Labs, 10Tool-Labs, 10Monitoring, 06Operations: Make icinga-wm report Tools homepage check at #wikimedia-labs, too - https://phabricator.wikimedia.org/T128716#2333567 (10valhallasw) p:05High>03Low [12:49:30] Gah. [13:02:27] 06Labs, 10Tool-Labs, 10WLX-Jury: Figure out a way to support java 1.8 on tool labs (For WLX Jury) - https://phabricator.wikimedia.org/T124903#2333594 (10Base) WMUK? [13:03:23] 06Labs, 10Tool-Labs, 10WLX-Jury: Figure out a way to support java 1.8 on tool labs (For WLX Jury) - https://phabricator.wikimedia.org/T124903#2333601 (10valhallasw) Sorry, WMUA (Wikimedia Ukraine). [13:04:33] 06Labs, 10Tool-Labs: Applebot overloading all of tool labs - https://phabricator.wikimedia.org/T134591#2333603 (10valhallasw) 05Open>03Resolved a:03valhallasw Banned (https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Banned); @Andrew emailed them but I don't think we have received a response, so applebo... [13:04:48] 06Labs, 10Tool-Labs, 13Patch-For-Review: Misleading "release is deprecated and has no effect, using trusty" - https://phabricator.wikimedia.org/T135349#2333607 (10valhallasw) 05Open>03Resolved a:03valhallasw [13:04:58] 06Labs, 10Tool-Labs, 13Patch-For-Review: Misleading "release is deprecated and has no effect, using trusty" - https://phabricator.wikimedia.org/T135349#2296004 (10valhallasw) a:05valhallasw>03yuvipanda [13:09:09] 06Labs, 10Tool-Labs, 13Patch-For-Review: Block xtools-articleinfo from scrapers - https://phabricator.wikimedia.org/T122327#2333619 (10valhallasw) p:05Triage>03Low [13:09:24] 06Labs, 10Tool-Labs: tool labs: provide custom domain proxy? - https://phabricator.wikimedia.org/T122403#2333621 (10valhallasw) p:05Triage>03Lowest [13:10:30] 06Labs, 10Tool-Labs: No API login with login token possible at tools-bastion-02 only - https://phabricator.wikimedia.org/T134262#2333628 (10valhallasw) 05Open>03Invalid >>! In T134262#2282183, @doctaxon wrote: > @bd808 simply except users with botflag from using a captcha to login That would beat the purp... [13:14:10] 06Labs, 10Tool-Labs: investigate webservices crash 28 dec 2015 - https://phabricator.wikimedia.org/T122509#2333663 (10valhallasw) 05Open>03Resolved a:03valhallasw I'm reasonably convinced {T122515} is the culprit, and 5 months later there is little investigation that can still be done. [13:14:21] 06Labs, 10Tool-Labs: Possible race condition in webservice HSET/HDEL - https://phabricator.wikimedia.org/T122515#1906301 (10valhallasw) p:05Triage>03High [13:15:08] 06Labs, 10Tool-Labs: Webservicemonitor doesn't understand webservice-new - https://phabricator.wikimedia.org/T122647#2333673 (10valhallasw) 05Open>03Resolved a:03yuvipanda This should be resolved now that T98440 is deployed. [13:16:30] 06Labs, 10Tool-Labs, 10WLX-Jury: Figure out a way to support java 1.8 on tool labs (For WLX Jury) - https://phabricator.wikimedia.org/T124903#2333685 (10intracer) Regarding the previously communicated statement that k8s is going to be stable by end of May, did the plans change or you encounter unexpected pr... [13:16:33] 06Labs, 10Tool-Labs, 10Pywikibot-core: Tool Labs: shared Pywikibot code not available - https://phabricator.wikimedia.org/T125505#1989468 (10valhallasw) 05Open>03Resolved a:05scfc>03None [13:22:21] 06Labs, 10Tool-Labs, 10WLX-Jury: Figure out a way to support java 1.8 on tool labs (For WLX Jury) - https://phabricator.wikimedia.org/T124903#2333733 (10yuvipanda) It is stable atm (https://tools.wmflabs.org/nagf/ is running on it). I'm currently working on making changes to the webservice tool to make it ea... [13:31:09] !log ores adding role::deployment::server to ores project [13:31:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL, Master [13:31:59] 06Labs, 10Tool-Labs, 10WLX-Jury, 13Patch-For-Review: Figure out a way to support java 1.8 on tool labs (For WLX Jury) - https://phabricator.wikimedia.org/T124903#2333768 (10yuvipanda) The only thing I'd need is that you provide me with a commandline that when run starts a webserver serving your application... [13:32:18] 06Labs, 06Operations, 06Release-Engineering-Team, 10wikitech.wikimedia.org: Rename specific account in LDAP, Wikitech and Gerrit - https://phabricator.wikimedia.org/T133968#2333770 (10lfschenone) [13:34:34] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 07Tracking: Goal: Allow using k8s instead of GridEngine as a backend for webservices (tracking) - https://phabricator.wikimedia.org/T129309#2333778 (10yuvipanda) tools.wmflabs.org/nagf is now running using the desired deployment process. I did it by hand, so that nee... [13:37:13] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 07Tracking: Goal: Allow using k8s instead of GridEngine as a backend for webservices (tracking) - https://phabricator.wikimedia.org/T129309#2333782 (10yuvipanda) When done, to move to k8s, a tool author would need to simply do: ``` webservice --backend=gridengine st... [13:43:32] !log ores-staging deleting ores-scaptest-01 [13:43:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores-staging/SAL, Master [13:53:06] andrewbogott: hey, if you are around. I can't connect to ores-staging-deployment.ores-staging.eqiad.wmflabs I made it a few minutes ago [13:53:26] "Permission denied (publickey)." [13:53:37] Amir1: I'll look — it may just take another few minutes to come up [13:54:02] okay [13:54:04] thanks :) [13:54:19] i'll let you know If it comes up [13:58:58] Amir1: what is your username on wikitech? [13:59:08] Ladsgroup [13:59:20] still can't connect [14:00:45] Amir1: for that instance (or maybe the whole project) the hiera config is broken [14:00:50] so it's unable to set up new instances. [14:00:51] [1;31mError: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find data item mediawiki_memcached_servers in any Hiera data file and no default supplied at /etc/puppet/modules/mediawiki/manifests/nutcracker.pp:19 on node ores-staging-deployment.ores-staging.eqiad.wmflabs[0m [14:01:17] hmm [14:01:22] thanks, let me check [14:01:24] You can see things like that in the 'Log' tab on horizon [14:01:48] Amir1: are you trying to setup a scap master? [14:02:03] YuviPanda: a tin instance [14:02:06] right [14:02:13] so that is 100% tied to mediawiki and is a big clusterfuck [14:02:19] I don't have any solutions, only warnings :) [14:02:35] proceed carefully, and good luck :) [14:02:45] andrewbogott: thanks. I check [14:03:15] YuviPanda: yeah, I know, maybe I just need to do an npm server instead of the role [14:03:19] I did that before [14:03:35] :) ok! [14:03:36] *nodejs server [14:03:52] thanks for the update [14:08:05] 10Tool-Labs-tools-Other, 10xTools-on-Labs: wikiviewstats webservice crashing all the time - https://phabricator.wikimedia.org/T122506#1906100 (10MusikAnimal) I don't think this tool is going to be revived [14:12:01] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 07Tracking: Allow accessing kubernetes services / apiserver from bastions - https://phabricator.wikimedia.org/T136413#2333910 (10yuvipanda) [14:12:07] valhallasw`cloud: ^ [14:12:57] that's... more complicated than I had thought [14:15:58] valhallasw`cloud: it just needs kubectl if it wants to only access APIServer [14:16:06] valhallasw`cloud: it requires the rest if it wants to hit service IPs / pod IPs [14:21:37] (03PS3) 10Yuvipanda: Block xtools-articleinfo from scrapers [labs/toollabs] - 10https://gerrit.wikimedia.org/r/291214 (https://phabricator.wikimedia.org/T122327) (owner: 10Merlijn van Deen) [14:21:47] (03CR) 10Yuvipanda: [C: 032] Block xtools-articleinfo from scrapers [labs/toollabs] - 10https://gerrit.wikimedia.org/r/291214 (https://phabricator.wikimedia.org/T122327) (owner: 10Merlijn van Deen) [14:36:13] PROBLEM - SSH on tools-webgrid-lighttpd-1408 is CRITICAL: Server answer [14:39:40] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 13Patch-For-Review, 07Tracking: Allow accessing kubernetes services / apiserver from bastions - https://phabricator.wikimedia.org/T136413#2333987 (10yuvipanda) I've also uploaded our flannel package to the wikimedia trusty repo. [14:45:05] !log tools start moving tools-bastion-03 to use tools-puppetmaster-01 as puppetmaster [14:45:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [15:04:35] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Documentation: Create a "my first Python webservice" tutorial for Tool Labs - https://phabricator.wikimedia.org/T134494#2334102 (10bd808) 05Resolved>03Open a:05valhallasw>03None I think that @valhallasw's existing tutorial is a great start, but it'... [15:04:38] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 06Developer-Relations, 07Documentation: Run a documentation sprint for Labs - https://phabricator.wikimedia.org/T101659#2334105 (10bd808) [15:04:40] 10Tool-Labs-tools-Other, 07I18n: [[Wikimedia:Pageviews-elapsed-time/en]] needs PLURAL - https://phabricator.wikimedia.org/T136300#2334108 (10Liuxinyu970226) [15:06:46] 10Tool-Labs-tools-Other, 07I18n: [[Wikimedia:Pageviews-num-pages/en]] needs PLURAL - https://phabricator.wikimedia.org/T136299#2334117 (10Liuxinyu970226) [15:11:03] valhallasw`cloud: \o/ thanks for all the task cleanup [15:12:00] 10Tool-Labs-tools-Other, 07I18n: [[Wikimedia:Pageviews-custom-range/en]] i18n issue - https://phabricator.wikimedia.org/T136297#2334124 (10Liuxinyu970226) [15:17:58] bd808: wait, I closed that task? O_o. [15:18:21] that was not my intention, but I've been fighting with phab a lot today [15:18:27] no worries [15:21:57] 10Tool-Labs-tools-Other, 10xTools-on-Labs: wikiviewstats webservice crashing all the time - https://phabricator.wikimedia.org/T122506#2334140 (10Matthewrbowker) p:05Triage>03High a:03Matthewrbowker I'll give it a shot, no promises though. [15:27:56] 06Labs, 10Tool-Labs: Prevent overly-large log files - https://phabricator.wikimedia.org/T122508#2334151 (10chasemp) Thanks @valhallasw Anything that is reactive here doesn't satisfy the task in my mind. If we setup a cron that finds and truncates, or compresses, or rotates it changes the nature of the proble... [16:27:06] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 13Patch-For-Review: Allow accessing kubernetes services / apiserver from bastions - https://phabricator.wikimedia.org/T136413#2334333 (10Danny_B) [16:36:22] 06Labs, 10Tool-Labs, 07Tracking: Overhaul logging setup for Tools (Tracking) - https://phabricator.wikimedia.org/T127367#2334393 (10valhallasw) [16:37:55] 06Labs, 10Tool-Labs, 07Tracking: Overhaul logging setup for Tools (Tracking) - https://phabricator.wikimedia.org/T127367#2041636 (10valhallasw) [16:38:40] 06Labs, 10Tool-Labs, 07Tracking: Overhaul logging setup for Tools (Tracking) - https://phabricator.wikimedia.org/T127367#2334402 (10bd808) [16:47:59] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 06Community-Tech-Tool-Labs: Develop evaluation criteria for comparing PaaS solutions - https://phabricator.wikimedia.org/T136265#2334436 (10bd808) Not sure what the explicit criteria is exactly, but the PaaS should have some tooling/integration/support for debug log... [17:05:19] 06Labs, 10MediaWiki-extensions-OATHAuth, 06Security-Team, 10wikitech.wikimedia.org, 13Patch-For-Review: wikitech 2fa provisioning form does so without confirmation - https://phabricator.wikimedia.org/T130892#2334533 (10csteipp) @dpatrick, sounds good [17:12:38] are there precise machines machines in toollabs? [17:12:49] eh, precise instances [17:15:52] mutante: yes, about 30% of them [17:16:01] mutante: tools-*-12?? [17:16:15] all other hosts are trusty or jessie [17:16:20] valhallasw`cloud: ok, thanks. i was wondering if some puppet code could be removed already that was just for precise [17:19:34] we are in some phase of moving off precise but I don't have a clear state of it mutante [17:22:58] chasemp: *nod* thanks, yep [17:40:28] 06Labs, 10Tool-Labs, 07Tracking: Tools that should get deleted (tracking) - https://phabricator.wikimedia.org/T133777#2334625 (10Danny_B) [17:43:53] 06Labs, 07Tracking: Create labtest cluster (tracking) - https://phabricator.wikimedia.org/T120293#2334648 (10Danny_B) [17:44:30] 06Labs, 10Tool-Labs, 07Tracking: Tool labs admin guides (tracking) - https://phabricator.wikimedia.org/T104734#2334649 (10Danny_B) [18:01:19] 06Labs, 10Wikimedia-Labs-General, 10DBA, 06Operations, 07Tracking: Database replication services (tracking) - https://phabricator.wikimedia.org/T50930#2334763 (10Danny_B) [18:28:14] 06Labs, 10wikitech.wikimedia.org: Two factor authentication unexpectedly enabled itself for Gifti - https://phabricator.wikimedia.org/T130953#2334832 (10dpatrick) [18:28:16] 06Labs, 10MediaWiki-extensions-OATHAuth, 06Security-Team, 10wikitech.wikimedia.org, 13Patch-For-Review: wikitech 2fa provisioning form does so without confirmation - https://phabricator.wikimedia.org/T130892#2334830 (10dpatrick) 05Open>03Resolved [18:35:29] 06Labs, 10Tool-Labs: Prevent overly-large log files - https://phabricator.wikimedia.org/T122508#2334858 (10scfc) >>! In T122508#2333201, @valhallasw wrote: > […] >> to being fine with unrecoverable errors above a certain threshold? > This is not different from quota, where you would get 'Disk quota exceeded'.... [18:57:31] !log toolsbeta sudo qconf -Ae /var/lib/gridengine/etc/exechosts/toolsbeta-exec-1209.toolsbeta.eqiad.wmflabs [18:57:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL, Master [18:58:10] yay, toolsbeta exec hosts are misconfigured [18:58:49] 06Labs, 10Tool-Labs: toolsbeta grid misconfigured - https://phabricator.wikimedia.org/T136433#2334933 (10valhallasw) [19:04:06] 06Labs, 10Tool-Labs: toolsbeta grid misconfigured - https://phabricator.wikimedia.org/T136433#2334952 (10valhallasw) Created host group `@general` using ``` valhallasw@toolsbeta-master:~$ sudo qconf -ahgrp group_name @general hostlist toolsbeta-exec-1209 ``` Copied the `task` queue definition from tools: `... [19:09:38] 06Labs, 10Tool-Labs: toolsbeta grid misconfigured - https://phabricator.wikimedia.org/T136433#2334964 (10valhallasw) ``` valhallasw@toolsbeta-master:~$ tail /var/spool/gridengine/qmaster/messages ... 05/27/2016 19:06:03| timer|toolsbeta-master|E|error opening file "/var/lib/gridengine/default/common/accounting... [19:25:20] 06Labs, 10Tool-Labs: Prevent overly-large log files - https://phabricator.wikimedia.org/T122508#2335041 (10valhallasw) It is in no way a perfect solution, but it is a solution that prevents the largest issues with a minimum of engineering effort. Having a better logging solution is on the radar (T127367). Fo... [21:07:24] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/ArthurPSmith was created, changed by ArthurPSmith link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/ArthurPSmith edit summary: Created page with "{{Tools Access Request |Justification=wikidata support - currently there is an issue with formatter URL's for external ID properties in wikidata, where previously a gadget ref..." [21:10:22] i once wrote down to move a database "to c2" [21:10:33] but i dont know much more :p [21:11:13] is it recommended to use that instead of installing mysql/mariadb at all on instances? [21:11:52] i dont even have a problem with the current setup, so maybe i just forget about touching it [21:19:56] 06Labs, 06Operations, 10Wikimedia-Video, 07Need-volunteer: Upload the Wikimania 2014 videos to Commons - https://phabricator.wikimedia.org/T106038#2335362 (10Dzahn) [21:20:47] 06Labs, 06Operations, 10Wikimedia-Video, 07Need-volunteer: Upload the Wikimania 2014 videos to Commons - https://phabricator.wikimedia.org/T106038#1457394 (10Dzahn) removed outdated "site-requests" tag. added Need-volunteer and Labs for attention and because the files are there. [21:22:39] hmm, puppet run hangs forever (to all purposes) after "Debug: /Stage[main]/Role::Labs::Nfsclient/Labstore::Nfs_mount[project]/File[/etc/idmapd.conf]: Nothing to manage: no ensure and the resource doesn't exist" [21:23:14] mutante: c2.labsdb (=labsdb1002) is one of the replica servers; you should really only use that for replicas or tables that need to join with replicas [21:23:58] MaxSem: there were some nfs issues last week, so maybe the mount is borked somehow? [21:24:00] valhallasw`cloud: yea, this was completely not related to wiki dbs from cluster, just a project in labs that uses mysql on localhost [21:24:15] valhallasw`cloud: apparently i thought once i should use a central db server instead [21:24:28] mutante: local mysql is a better idea than c2, and if you want to use a central db, please use toolsdb instead [21:24:49] valhallasw`cloud: what if the project is in labs but not toollabs [21:25:05] valhallasw`cloud: i am fine with staying local :) [21:25:54] mutante: toolsdb is a bit misnamed, it's just one of the shared database servers, but without replicas [21:26:08] valhallasw`cloud: ah! ok, thanks [21:28:12] mutante: where does that puppet run hang? what host? [21:29:56] chasemp: i think you meant MaxSem [21:29:58] MaxSem: ^ [21:30:32] chasemp, maps-tiles{1,2..} [21:31:05] interestingly, Ctrl+C doesn't abort it, only kill -9 does [21:34:10] 10Labs-project-wikistats: move wikistats database to c2 - https://phabricator.wikimedia.org/T110391#2335394 (10Dzahn) 14:29 < valhallasw`cloud> mutante: local mysql is a better idea than c2, and if you want to use a central db, please use toolsdb instead 14:31 < valhallasw`cloud> mutante: toolsdb is a bit misna... [21:34:22] MaxSem: that sounds like NFS trouble [21:34:28] 10Labs-project-wikistats: move wikistats database to toolsdb - https://phabricator.wikimedia.org/T110391#2335395 (10Dzahn) [21:35:35] oops :) [21:35:44] MaxSem: I bet it's related to /public/dumps shenangians [21:35:58] you can try fuser -k(m?) /public/dumps and unmount and remount [21:36:05] it may take a restart depending on release [21:36:47] mhm, maybe just restart? :P [21:43:33] :) [21:45:34] w00t, works [21:45:46] the offendig thingie appears to be Exec[block-for-nfs-home](provider=posix): Executing check '/bin/mountpoint -q /home' [21:52:01] !log maps Rebooted maps-tiles1, maps-tiles2 and maps-warper due to NFS troubles [21:52:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Maps/SAL, Master [21:53:34] maps-warper appeared to be okay re puppet earlier [21:53:53] I had puppet failure spam from it too [21:54:17] there was some issue, but I think it got resolved [21:55:19] also, EVEN WINDOWS SHARES WORK BETTER THAN NFS!!1 XD [21:55:25] heh [21:55:50] at least, timeouts are less eternal [21:59:22] I thought it could have also been due to the issues of labvirt1003, at least I think after that was resolved I didnt get the email about the instance (but I did continue to get ones about tiles) [22:20:28] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 13Patch-For-Review: Allow accessing kubernetes services / apiserver from bastions - https://phabricator.wikimedia.org/T136413#2335584 (10yuvipanda) a:03yuvipanda It's now running flannel and kube-proxy properly! \o/ [23:32:24] Is anyone here experienced with python multi-threading and interested in helping me play with my new raspberry pi? [23:34:12] TParis: it's been pretty quiet this afternoon. everyone is lurking or something [23:34:36] bd808: I see okay [23:34:45] bd808: Is 808 your area code? [23:36:22] nope. the bd808 handle was given to my by internic when I registeredc my first domain. I was the 808th person with the initials BD to be associated with a domain [23:36:46] i used it on and off for years when I couldn't get my prefered nick on a site [23:36:51] Ohh okay, I see. I thought we might've been neighbors. [23:37:01] finally I just decided that it would work everywhere [23:37:10] I'm a 208 area code :)