[00:01:58] Our two repos are both managed by the same puppet class. So it's possible that they are corrupt in the same way via the same process. I'll poke around a bit. [00:02:52] PROBLEM Total processes is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS WARNING: 160 processes [00:06:53] PROBLEM Free ram is now: WARNING on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: Warning: 19% free memory [00:10:24] Change on 12mediawiki a page Wikimedia Labs/Account creation improvement project was modified, changed by 216.38.130.166 link https://www.mediawiki.org/w/index.php?diff=630710 edit summary: [-66] /* Current account creation process */ [00:11:45] Change on 12mediawiki a page Wikimedia Labs/Account creation improvement project was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=630711 edit summary: [-115] /* Current account creation process */ [00:37:53] RECOVERY Free ram is now: OK on swift-be3.pmtpa.wmflabs 10.4.0.124 output: OK: 21% free memory [00:41:52] RECOVERY Free ram is now: OK on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: OK: 20% free memory [00:42:52] RECOVERY Total processes is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS OK: 150 processes [00:42:53] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 20% free memory [00:48:40] Wikinaut: Here's someone with the same issue. http://comments.gmane.org/gmane.comp.version-control.git/182469 I don't really understand what they mean by 'removing all refs from the remote' though [00:58:53] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 8% free memory [01:05:52] PROBLEM Free ram is now: WARNING on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Warning: 18% free memory [01:08:42] PROBLEM Total processes is now: WARNING on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS WARNING: 184 processes [01:09:52] PROBLEM Free ram is now: WARNING on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: Warning: 17% free memory [01:13:43] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 107 processes [01:21:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 6.10, 6.11, 5.19 [01:21:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 7.83, 6.40, 5.33 [01:26:43] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 2.70, 4.42, 4.77 [01:26:53] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 3.39, 4.71, 4.91 [01:39:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 9.06, 7.38, 5.81 [01:39:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 11.32, 8.75, 6.60 [01:45:44] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 8.52, 7.57, 5.82 [03:03:54] PROBLEM Total processes is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS WARNING: 160 processes [03:04:16] wooah [03:04:17] 160 [03:04:17] addshore ... [03:04:21] what have you DONE [03:20:09] Vacation9, not much :O [03:20:27] i only have 14 procs [03:20:42] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: OK - load average: 3.28, 4.26, 4.90 [03:21:02] and 3 of them are me logged in and another runnign the command to find out so really only 10 [03:28:54] RECOVERY Total processes is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS OK: 146 processes [03:34:45] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 3.89, 3.95, 4.64 [03:41:52] PROBLEM Total processes is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS WARNING: 153 processes [03:42:18] @_@ Not again [03:54:52] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 3.53, 4.21, 4.75 [04:38:52] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 28% free memory [04:39:52] RECOVERY Free ram is now: OK on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: OK: 21% free memory [04:40:52] RECOVERY Free ram is now: OK on swift-be3.pmtpa.wmflabs 10.4.0.124 output: OK: 23% free memory [04:41:52] RECOVERY Total processes is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS OK: 150 processes [04:47:53] PROBLEM Free ram is now: WARNING on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: Warning: 17% free memory [04:48:53] PROBLEM Free ram is now: WARNING on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Warning: 18% free memory [05:06:53] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 15% free memory [05:38:52] RECOVERY Free ram is now: OK on swift-be3.pmtpa.wmflabs 10.4.0.124 output: OK: 35% free memory [05:42:31] petan, would labs be a sensible place to setup prototype wiki for a proposal to wikimedia? http://meta.wikimedia.org/wiki/Wikissentials [05:42:44] Or would I be better of just using http://www.wikia.com ? [05:57:56] Ryan_Lane, :) Just the person! would labs be a sensible place to setup prototype wiki for a proposal to wikimedia? http://meta.wikimedia.org/wiki/Wikissentials Or would I be better of just using http://www.wikia.com ? [06:29:55] PROBLEM Total processes is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS WARNING: 152 processes [06:32:31] Addshore-w go sleep [06:32:38] it's 6 am [06:35:32] shhh [06:38:57] labs is a place to do anything you want, but don't tell to others... [06:39:10] it's secret [06:45:13] ;p [06:45:34] petan, is there a way to make wm-bot watch my user pages on all wikis? [06:47:09] not yet [06:47:22] :< [06:49:54] RECOVERY Total processes is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS OK: 148 processes [07:04:54] petan, can you add wikidata to the rchanges for wm-bot? :) [08:00:00] Addshore-w it's [08:00:53] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 29% free memory [08:01:02] ahh got it [08:41:52] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 28% free memory [08:49:52] PROBLEM host: mobile-osm2.pmtpa.wmflabs is DOWN address: 10.4.1.67 CRITICAL - Host Unreachable (10.4.1.67) [08:53:52] RECOVERY host: mobile-osm2.pmtpa.wmflabs is UP address: 10.4.1.67 PING OK - Packet loss = 0%, RTA = 0.64 ms [08:54:22] PROBLEM Total processes is now: CRITICAL on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: Connection refused by host [08:54:52] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 8% free memory [08:55:53] PROBLEM Current Load is now: CRITICAL on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: Connection refused by host [08:55:53] PROBLEM dpkg-check is now: CRITICAL on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: Connection refused by host [08:56:33] PROBLEM Current Users is now: CRITICAL on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: Connection refused by host [08:57:13] PROBLEM Disk Space is now: CRITICAL on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: Connection refused by host [08:58:04] PROBLEM Free ram is now: CRITICAL on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: Connection refused by host [09:03:02] RECOVERY Free ram is now: OK on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: OK: 899% free memory [09:04:22] RECOVERY Total processes is now: OK on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: PROCS OK: 84 processes [09:05:45] Ryan_Lane: hi [09:05:53] RECOVERY Current Load is now: OK on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: OK - load average: 0.05, 0.59, 0.54 [09:05:53] RECOVERY dpkg-check is now: OK on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: All packages OK [09:06:35] RECOVERY Current Users is now: OK on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: USERS OK - 0 users currently logged in [09:07:13] RECOVERY Disk Space is now: OK on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: DISK OK [09:33:44] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.87, 4.89, 4.97 [09:46:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 5.06, 5.08, 5.05 [09:51:43] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.79, 4.89, 4.98 [10:47:53] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 27% free memory [11:19:54] PROBLEM Free ram is now: CRITICAL on sube.pmtpa.wmflabs 10.4.0.245 output: Critical: 3% free memory [11:24:53] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 8% free memory [11:32:52] PROBLEM Total processes is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS WARNING: 153 processes [11:47:53] RECOVERY Total processes is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS OK: 149 processes [12:40:23] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 20% free memory [12:46:07] !log bots petrb: disabling sql3 in order to change the device [12:46:09] Logged the message, Master [12:57:32] !log bots petrb: moving the data to gluster until I recreate filesystem [12:57:34] Logged the message, Master [13:05:23] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 13% free memory [13:09:47] !log bots root: scheduling sql3 to reboot [13:09:48] Logged the message, Master [13:10:10] !log bots root: sql3 will reboot approximately in 1h [13:10:11] Logged the message, Master [13:46:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 4.95, 5.03, 5.01 [14:06:52] PROBLEM Total processes is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS WARNING: 153 processes [14:16:54] RECOVERY Total processes is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS OK: 149 processes [14:47:43] PROBLEM Free ram is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Critical: 5% free memory [14:50:47] !log bots petrb: sql3 done [14:50:48] Logged the message, Master [15:24:44] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 4.88, 5.04, 5.01 [15:25:03] Ryan_Lane: listening? [15:34:43] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.96, 4.95, 4.99 [16:02:32] PROBLEM Free ram is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: Warning: 14% free memory [16:37:37] Ryan_Lane, about? :) [16:39:52] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 27% free memory [16:40:22] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 20% free memory [16:48:23] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 19% free memory [17:03:02] PROBLEM dpkg-check is now: UNKNOWN on sube.pmtpa.wmflabs 10.4.0.245 output: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [17:07:53] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 15% free memory [17:08:03] PROBLEM dpkg-check is now: CRITICAL on sube.pmtpa.wmflabs 10.4.0.245 output: DPKG CRITICAL dpkg reports broken packages [17:43:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 4.76, 4.97, 5.01 [17:47:37] !log bots addshore: updating libfreetype6, libnspr4-0d, libnss3-1d [17:47:39] Logged the message, Master [17:48:43] !log bots addshore: bots-4 updating libfreetype6, libnspr4-0d, libnss3-1d [17:48:44] Logged the message, Master [18:04:12] PROBLEM Disk Space is now: UNKNOWN on sube.pmtpa.wmflabs 10.4.0.245 output: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [18:09:14] PROBLEM Disk Space is now: CRITICAL on sube.pmtpa.wmflabs 10.4.0.245 output: DISK CRITICAL - free space: / 0 MB (0% inode=38%): [20:37:52] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 27% free memory [20:38:22] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 23% free memory [20:52:33] PROBLEM Free ram is now: CRITICAL on bots-4.pmtpa.wmflabs 10.4.0.64 output: Critical: 1% free memory [20:55:52] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 10% free memory [21:01:23] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 14% free memory [21:02:33] RECOVERY Free ram is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: OK: 197% free memory [21:23:43] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 5.19, 4.96, 5.00 [21:31:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 5.25, 5.14, 5.07 [21:44:20] OpenID users here ? [21:44:30] Wikinaut: eh? [21:44:48] openid-wiki.instance-proxy.wmflabs.org : up and running w/ OpenID w/o a single problem [21:45:04] latest versions core, extensions [21:45:10] :-) [21:45:24] what about the open bugs for acting as a forced provider? [21:45:30] he slowly [21:45:36] what about my email to you ? [21:45:41] which email? [21:46:13] there's also an open bug with the openid provider using its own login form [21:46:20] labs instance login problem with a second public key 09:43 UTC [21:47:02] Ryan_Lane: regarding the bug report: I had not any single problem with my own wikis [21:47:05] but of course [21:47:13] I will try to fix the filed issues [21:47:33] well, we want to integrate other apps with labsconsole [21:47:47] so we want to make those other apps use openid as a consumer [21:47:53] mom [21:47:56] but we want to force the provider from the other app [21:50:13] WOW! it works [21:50:47] eh? what does? [21:50:58] you can try it. You can use your userpage http://openid-wiki.instance-proxy.wmflabs.org/wiki/User:XXXXXX (MUST have content!!!) as OpenID elsewhere [21:51:12] .... [21:51:17] the userpage be non-empty [21:51:37] you can't use that to force provider usage [21:51:50] with a forced provider the user doesn't enter any information [21:52:02] they just click "log in", then they are redirected to the wiki [21:52:08] I need more details [21:52:10] which will figure out which user they are [21:52:11] tell me SLOWLY [21:52:16] this is slowly [21:52:17] what you mean exactly [21:52:22] he [21:52:24] there's an open bug report on this [21:52:33] man [21:52:44] I spent 500 hours on that [21:52:48] bear with me [21:53:00] https://bugzilla.wikimedia.org/show_bug.cgi?id=40068 [21:53:14] the current feature is fine if you always expect a user to provide a url to log in [21:53:32] I will look into this. Promised [21:53:36] but most consumers want to provide a button you can click [21:53:41] Please read my mail re. the root [21:54:03] public key authentication to the instance [21:54:15] don't use more than one key [21:54:19] why would you need to? [21:54:28] explained in my mail [21:54:29] you shouldn't be logging in directly as root [21:54:30] pls. read [21:54:39] not logging in .. [21:54:42] but for WinSCP [21:54:48] that's the same thing [21:54:52] hehe [21:54:54] read my mail [21:54:56] pls [21:54:58] I did [21:55:05] that's why I'm asking you [21:55:10] write files as your own user [21:55:15] pls answer Q1 and Q2 [21:55:18] of my mail [21:55:28] why? the answer is: don't do that [21:55:43] don't directly copy files as root [21:55:46] it's bad practice [21:55:48] Ryan_Lane: yes [21:55:54] pls. answer Q2 [21:56:02] dude. [21:56:05] just that I know [21:56:07] the answer is simple [21:56:11] don't do it [21:56:14] man [21:56:26] I want to know WHY it does not work [21:56:37] I do not push you to change that [21:56:47] I'm asking you not to do it [21:56:58] Q2: Has it to to with the fact that the VMs are set up with LDAP, and [21:56:59] this only accepts the stored keys in labs/gerrit ? [21:57:02] but otherwise I'd actually need to go on to the system to figure out why [21:57:18] and I don't see the point in troubleshooting something you shouldn't be doing [21:57:21] it is unfriendly not to answer kind questions [21:57:44] answering your question would require me to log into your instance and see what you did wrong [21:57:50] ok [21:57:58] but if you have time, I am interested in that [21:58:02] as you could see [21:58:09] I managed to install [21:58:20] the files, git pull etc. [21:58:22] I get that. this is one of those rare times I'll have to say figure it out on your own [21:58:29] by using the correct method [21:58:29] but please don't add root keys [21:58:36] ok [21:58:41] your changes will get overwritten anyway [21:58:46] I *do* understand [21:59:08] perhaps... [21:59:14] I'm glad you have open id set up [21:59:24] it'll be a good place to work together on bugs for the extension [21:59:27] it worked immediately [21:59:34] well, of course [21:59:43] no one is implying the extension is broken for its current feature set [21:59:45] I just followed strictly my own documentation on the E:OpenID page [22:00:15] It's a pleasure for me top have this official instance. Thanks [22:00:27] I am looking forward to work with you... [22:00:39] that OpenID is AuthPlugin compliant [22:00:49] you filed something, that it is currently not [22:00:56] We together should fix that [22:01:02] pls. help me [22:01:06] if I need help [22:01:08] ty [22:02:12] Ryan_Lane: is it possible (for "normal VMs I do know, how) to change from http:// to https:// [22:02:29] Change on 12mediawiki a page Wikimedia Labs/Account creation improvement project was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=631071 edit summary: [+884] /* Current account creation process */ [22:02:41] Wikinaut: not currently [22:02:42] I mean to https://openid-wiki.instance-proxy.wmflabs.org [22:02:45] ok. [22:02:51] pls. let me, when it is [22:02:57] we have plans to add https to the reverse proxy [22:03:04] I run all my wikis only via https:// [22:03:11] since about 1 year [22:03:14] yeah, we have a preference for that too [22:03:44] https:// via reverse proxy is (in normal systems) easy. I am pretty sure, you are aware of this [22:04:04] seeing as that I made the one for wikipedia? yes ;) [22:04:29] we need to control access to the ssl certificate [22:04:51] the current reverse proxy is a project that a volunteer has access to [22:05:00] for test instances, this isn't big issue. Or? [22:05:10] for example, I am member of CAcert [22:05:10] which would mean that person could take the cert/key and MITM services [22:05:26] MITM: arrrgh [22:05:29] we'd get a * certificate [22:05:38] uh [22:05:40] I'd much prefer that we don't lose a * cert/key [22:05:43] *.google.com [22:05:49] TürkCA [22:05:57] DigiNotar... [22:06:09] Change on 12mediawiki a page Wikimedia Labs was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=631073 edit summary: [+87] /* Proposals */ [22:06:18] that doesn't really apply to our situation [22:06:43] mp, [22:07:23] (standby) [22:08:25] Ryan_Lane: SSL is far less secure than most people believe anyway :P [22:08:37] yes, but we shouldn't actively give away keys ;) [22:08:42] but still much better than nothing [22:10:08] * Damianz pats MaxSem [22:11:52] some reading https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2031409 [22:12:47] Damianz: http://www.mediawiki.org/wiki/Wikimedia_Labs/Account_creation_improvement_project#Current_account_creation_process [22:12:57] did you see the idea of "Open-to-all projects"? [22:13:33] Ryan_Lane: the VMs, are they backuped daily or when? [22:13:41] Wikinaut: they aren't backed up at all [22:13:45] hmmm [22:13:45] uh [22:13:57] any ideas for that ? [22:14:01] Wikinaut: if you wish to back something up, you should back it up to /data/project [22:14:11] documented where? [22:14:17] that could be interesting [22:14:23] Wikinaut: it's shared storage inside of your project that isn't linked to your instance [22:14:24] you mean, copying to that [22:14:26] we need to stop using shell in mw though as it's so limiting [22:14:33] Wikinaut: if your instance goes away, the data will still be there [22:14:36] ok [22:14:41] Damianz: I linked some bugs [22:14:51] not read yet, will do after this program finishes [22:14:56] Damianz: I'm going to have mediawiki sync groups, and also be able to modify them [22:15:01] shell will become a posix group [22:15:09] we'll require shell on every instance in every project [22:15:19] that's easy to do pam wise [22:15:45] then on bastion, and other Open-to-all projects we'll drop the requirement to be in the project group [22:16:09] yeah [22:16:21] Downside to this is we'll make project membership look funky in keystone [22:16:26] it's just pam security and access.conf, like before [22:16:40] Though in bastion we don't care about membership... so linux is fine [22:16:43] well, it'll mean that those projects won't have a membership list [22:16:51] Other projects we kinda care for to know who's active etc [22:16:53] except for project admins [22:16:55] indeed [22:17:07] I don't see /data/project on "mount", but it's there [22:17:13] It could make it more confusing or simplier hmmm [22:17:16] Wikinaut: it's an automount [22:17:20] Wikinaut: ls /data/project [22:17:22] should kill automount [22:17:25] then check mount [22:17:36] Damianz: why? it's actually a good use here [22:17:36] also we need a better signup workflow and ux, it sucks currently [22:17:39] Damianz: it's browsable now [22:17:46] it just doesn't show in mount [22:17:47] Ryan_Lane: Most the time it's mounted always anyway :P [22:17:49] until you access it [22:17:59] Damianz: you'd think so, but that's not really true [22:18:05] also... [22:18:11] it means we can control it without puppet [22:18:13] ... indeed. Now it's in.... [22:18:15] magic [22:18:32] ok... technically it will fake being mounted until something does something with the fhs [22:18:39] Damianz: so, for instance, when we moved the homedirs over, I didn't need to manually fix many instances [22:18:42] maybe like 30 [22:18:58] still needed to reboob a bunch [22:19:06] reboob? [22:19:20] `reboot` [22:19:29] actually, I didn't need to reboot some [22:19:37] 63TB available ... :-) [22:19:43] for many of them I just needed to restart autofs [22:19:46] (don't worry, i am not evil) [22:19:50] *shrug* [22:19:53] Wikinaut: you're quota'd to 300GB ;) [22:19:58] uhuhuh [22:20:15] you can request for the quota to be higher [22:20:27] I'm still interested in abstracting the data dir.... doing global data shares with project permissions/automount config would be sexy [22:20:37] ye [22:20:38] *yes [22:20:39] not today... [22:20:40] that would be nice [22:20:55] global data shares are hard, though [22:20:59] from a security POV [22:21:20] it seems like a really great way to project hop [22:21:24] global read is easy, but yeah.... rbac w/ projects would be really the way to go [22:21:35] indeed [22:21:47] it would be nice to say project x, y, and z can access this r/w [22:21:50] like bots and webtools [22:22:02] mhm [22:22:22] It would also be kinda nice to be able to create random groups [22:22:29] I think both bots and webtools could eventually be open-to-all [22:22:35] Ryan_Lane: my questions are answered so far. Do you have any for me? [22:22:47] like we can say 'share' is for in and are part of group and the perms are setup so they can access without r00t [22:22:49] Wikinaut: nope. just wondering about the two open bugs :) [22:22:58] Wikinaut: http://www.mediawiki.org/wiki/Wikimedia_Labs/Account_creation_improvement_project#OpenID_as_a_provider [22:23:04] bots needs per user process tracking before opening really [22:23:21] well, we definitely need to not allow root for it [22:23:31] Ah, i did not yet know that these .. [22:23:38] are blocking that [22:23:41] now I do know [22:23:53] yeah and to block root we need to do groups really [22:23:57] or we're back to per user stuff [22:24:06] do groups easily? [22:24:25] ideally only project admins would have root [22:24:33] and that can be managed via sudo policy [22:24:55] Change on 12mediawiki a page Wikimedia Labs/Account creation improvement project was modified, changed by Wikinaut link https://www.mediawiki.org/w/index.php?diff=631077 edit summary: [+120] /* OpenID as a provider */ +: ''accepted. Will do my best. --~~~~ [22:25:40] Wikinaut: well, it's best to not say you're going to take the bugs till you're ready to complete them immediately [22:25:51] others may do it before you get a chance [22:26:00] Change on 12mediawiki a page Wikimedia Labs/Account creation improvement project was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=631078 edit summary: [-120] /* OpenID as a provider */ [22:26:09] tyler, for instance, pushed in a change for one of the bugs [22:27:13] meh [22:27:19] also why accepted as a comment [22:27:23] there's stuff in bz for that [22:27:29] hate dispersed information [22:28:28] * Ryan_Lane nods [22:28:28] Ryan_Lane: I am not a cookie licker [22:28:34] but you can help me [22:28:45] I'm organizing bugs into projects, but otherwise all info is kept in BZ [22:28:46] Can you give a _concrete_ example on https://bugzilla.wikimedia.org/show_bug.cgi?id=40068 [22:28:59] sure [22:29:09] have you used openid as a consumer with google? [22:29:14] or launchpad? [22:29:22] *shudder* launchpad [22:29:33] pls. explain [22:29:39] when you login, you click a button that says "log in with google" or "log in with launchpad" [22:29:54] this works: google as provider, mediawiki as consumer [22:30:07] let me finish [22:30:51] when you visit those sites by clicking the button, they provide auto-discovery [22:31:01] you don't say "I want to log in with my open id url" [22:31:16] you are saying "here's the autodiscovery url of the server" [22:31:25] and it'll figure out what your openid url is itself [22:31:37] yes [22:31:44] the openid extension's provider doesn't support this [22:31:57] so, if I wanted to say "log in with labsconsole", I can't [22:32:05] Ah [22:32:15] users would need to provide their openid url [22:32:19] You want to have this button "Login with Labs" [22:32:24] yes [22:32:30] clear [22:32:42] (it is mentioned in the bug, I know) [22:32:48] but I wanted to be sure [22:33:07] well, Local Time Berlin is 23:33 [22:33:17] night is long [22:33:24] :-) [22:35:08] and this as button? https://labsconsole.wikimedia.org/w/images/thumb/6/65/Wikimedia_labs_small_logo.png/106px-Wikimedia_labs_small_logo.png [22:35:40] we'd have some button for consumers [22:35:48] but that wouldn't need to live in the mediawiki interface [22:36:06] this isn't mediawiki as a consumer of mediawiki. it's other applications as a consumer of mediawiki [22:36:28] but you want to login with LabsAccount ! [22:36:34] from other applications [22:36:42] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.43, 4.88, 4.99 [22:36:45] puzzled [22:36:53] to other apps [22:37:05] yes. other applications as a consumer of labs openid [22:37:09] yes [22:37:15] so you have ONE Labs Button [22:37:40] it doesn't need to be a button. I plan on forcing the consumer [22:37:51] so clicking on the login link of an application would make you use labs openid automatically [22:37:52] example? [22:37:57] for an consumer [22:38:01] pls [22:38:25] https://wiki-staging.openstack.org/wiki/Main_Page [22:38:33] let me summarize what I understand: [22:38:42] that wiki forces using launchpad as a provider [22:38:58] if you click on the login link, you are logged in via launchpad [22:39:19] so, the same idea, except the consumer would be something other than mediawiki [22:39:22] ok, and you want s/launchpad/Labs/ [22:39:32] labs would be a provider [22:39:37] yes [22:39:40] any application would be a forced consumer [22:40:13] I do not (yet) understand, what a "forced consumer" is [22:40:29] I guess, a hard-coded consumer [22:40:30] that wiki I linked you to was a forced consumer of launchpad [22:40:38] ok [22:40:42] the only openid it allowed was launchpad [22:40:47] yes [22:40:52] good [22:41:54] going to Labs (logging in, or later, during the authentication on the consumer) --> seeing a selected menu of consumers [22:42:05] no authent to others [22:42:10] ? [22:42:34] eh? [22:42:55] when you log into labs you use a username and password, and possibly a token [22:43:09] (or OpenID.... a special case later) [22:43:14] when you log into other applications inside of labs it would log you in via labsconsole [22:43:14] no openid [22:43:17] no openid [22:43:25] (it was in () ) [22:43:26] username, password, and possibly a token [22:43:30] yes [22:43:37] we can't currently support openid [22:43:48] as a consumer on labsconsole [22:43:54] It would be nice [22:43:58] it would. yes [22:44:03] Since then we could do sign in via central auth on labs [22:44:05] or such [22:44:06] that's not a small amount of work [22:44:26] OpenID must be Auth compliant, but is not [22:44:36] yeah.. you'd kinda need to make keystone do the oauth bit for tokens [22:44:48] yes. keystone doing oauth would fix this [22:44:55] that also means mediawiki would be able to do it too, though ;) [22:53:18] Damianz: here's something you may like: http://www.mediawiki.org/wiki/Wikimedia_Labs/Interface_usability_improvement_project [22:53:18] :) [22:53:48] Ryan_Lane, how easy is it to get OpenStackManager set up and at a point where you can develop these Echo notifications? [22:54:00] Krenair: we have a development instance for this [22:54:14] it takes a lot of related infrastructure to set up your own [22:54:26] but we can give you access to the project [22:54:45] are you interested in working on it? [22:55:05] I'm really excited about echo. it'll greatly improve things in labsconsole [22:55:12] Being completely unfamiliar with OpenStackManager, I can't promise anything [22:55:23] But I'd be happy to give it a try [22:55:34] well, a lot of these notifications would be triggered from maintenance scripts [22:55:45] except for instance deletions [22:56:12] for instance, when an instance reboots, OpenStackManager itself doesn't really know [22:56:31] but, salt will trigger an event on the instance when it comes back it [22:56:32] *up [22:56:53] when that occurs, the salt master can catch the event and run a maintenance script [22:57:03] which would trigger a notification [22:57:08] openstack should know... so triggers shouldn't be ard [22:57:16] I know literally nothing about salt. [22:57:21] Krenair: no need for that [22:57:28] if the maintenance script exists, I can handle the salt part [22:57:34] What I do know about is the Echo and MW maint script side of things. [22:57:47] yep [22:58:02] I can also show you how I'm handling the salt part, as well, if you're interested in learning it [22:58:23] See, I really am clueless about salt - what is it exactly? [22:58:34] Damianz: true, for reboots, it's possible to let openstack notify [22:58:41] It's not really something that's easy to google. [22:58:44] Krenair: saltstack http://saltstack.org/ [22:58:53] it's a remote execution framework [22:59:05] and configuration management [22:59:17] zeromq is smexy [22:59:37] and an asynchronous event driven programming framework [22:59:51] and yeah, it's based on zeromq [23:00:07] it's pretty awesome for lots of things [23:00:22] Ok, so this will run a system command which launches PHP and a MW maint script with some information about what has happened [23:00:30] yep [23:01:03] What I need to do is get that to create an Echo notification for the relevant people [23:01:14] indeed [23:02:21] why all sysadmins/netadmins btw, surly creator makes more sense [23:02:27] though a 'firehose' data stream would be fun [23:02:33] depends... [23:02:39] for reboots it may not have been the creator who did it [23:02:48] in that case it should be the rebooter [23:03:03] in the case of creations, it could just be the creator or all admins [23:03:09] mhm [23:03:11] deletions should likely be all admins [23:03:25] For creations I'd go with all admins... [23:03:31] agreed [23:03:41] it's nice to know when other people are creating resources in a project you manage [23:04:01] Damianz: things like echo are going to make it hard to ever switch to horizon :) [23:05:06] horizon is kinda ugly [23:05:22] I don't disagree, but it supports a lot more [23:05:33] though I guess it's nice to be able to control the rate of features [23:05:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 5.42, 5.32, 5.19 [23:05:50] adding a new feature to labsconsole takes less time than it takes to add it on the backend by far [23:06:07] it also is easier to break, horrible to write and is generally ugly [23:06:13] heh [23:06:29] well, OpenStackManager isn't great either ;) [23:07:39] hm [23:07:48] I wonder where to put this bug I'm creating [23:07:57] "Add Echo notification for new requests in queues" [23:08:05] it's really a SMW bug [23:09:53] Okay, so what information can we get salt to give us? The instance ID, and therefore the project and project's admins? And can I be added to this dev project? [23:10:07] yes to all of it [23:10:47] btw, if you need to upgrade MW for this, feel free to do so on the development instance [23:10:53] or if you'd like me to, I can as well [23:11:24] what's your labsconsole user name? [23:11:28] I probably won't need to... [23:11:30] Alex Monk [23:11:33] ok [23:12:10] hm [23:12:17] why is it failing to add your user? [23:12:26] oh [23:12:27] it didnt [23:12:41] ok. you're added [23:12:52] the instance name is nova-precise2 [23:14:04] wiki is under: /srv/org/wikimedia/controller/wikis/ [23:14:31] What is my SSH user name? Same as in Gerrit? [23:14:37] yep [23:15:46] Ok, so my key is already in Special:NovaKey [23:15:50] I should be able to do "ssh -A krenair@bastion.wmflabs.org"? [23:15:54] yep [23:16:21] Failed publickey for krenair [23:16:22] Don't you need to add me to the bastion project? [23:16:30] I wonder if we can convince Chad to fix the key bug [23:16:41] are you not in it? lemme check [23:17:08] Shouldn't be. I registered before this was default and never requested access to anything in labs. [23:17:11] ah [23:17:12] right [23:17:34] try now [23:17:40] works :) [23:17:43] great [23:18:25] Change on 12mediawiki a page Wikimedia Labs/Interface usability improvement project was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=631095 edit summary: [+117] /* Notifications */ [23:23:51] Okay, so let's see if I can work out where everything is [23:25:07] let me know if you have any questions [23:27:21] Ryan_Lane, Firstly do you think upgrading 208 packages on bots-3 will break anything? - Also how long do new labs projects generally take? :) [23:27:30] heh [23:27:46] new projects take? what do you mean? [23:28:00] or do you mean new instances in a project? [23:28:36] * Damianz thinks we should have a project that does leg massages [23:29:42] heh [23:29:59] I think it'll be a while before we get support for real-world projects in labsconsole [23:38:11] Ryan_Lane, so /data/project/core should be being served from nova-precise2.pmtpa.wmflabs:80? [23:38:50] I don't think so [23:39:00] /srv/org/wikimedia/controller/wikis/w should be [23:39:11] I believe [23:39:30] yep [23:39:32] Alias /w /srv/org/wikimedia/controller/wikis/w [23:39:32] Alias /wiki /srv/org/wikimedia/controller/wikis/w/index.php [23:39:49] Okay so I need to find a way to proxy my browser to nova-precise2.pmtpa.wmflabs somehow [23:40:22] socks proxy [23:40:32] there's also an instance proxy [23:40:39] hm [23:40:47] well, no, you want to use a socks proxy [23:40:55] since that's actually how we're handling that instance [23:41:10] !socks-proxy [23:41:10] ssh @bastion.wmflabs.org -D ; # [23:41:53] Socks proxy would require me to change chrome settings and would make all my requests go through bastion? [23:42:24] yes. it's also possible to use the instance proxy, but you'll need to change mediawiki's configuration to use the new address [23:43:27] oh [23:43:27] wai [23:43:28] wait [23:43:33] maybe not, because it uses https [23:43:42] * Ryan_Lane grumbles [23:43:42] Oh god, chrome is using ubuntu's network settings. Don't think I want all my traffic going through WMF servers :) [23:43:46] heh [23:43:53] yeah, that's a problem with chrome [23:44:00] using firefox it's possible to use foxyproxy [23:44:22] which will selectively send traffic via labs by url matching patterns [23:44:29] Will try that. [23:44:35] firefox is my pointed via a proxy browser [23:46:27] hey there's a foxyproxy for chrome. [23:46:35] last time I tried it it didn't work [23:46:40] it was broken [23:46:52] it was very frustrating :( [23:47:12] I should try it again one day [23:47:19] hm, yep :( [23:47:58] doesn't seem to be working for me [23:49:45] well, it partially works [23:49:51] the url pattern matching doesn't work [23:49:57] but, you can quickly switch between proxies [23:51:35] FF is completely broken for me at the moment. It's not loading any pages... [23:51:59] with or without foxyproxy? [23:52:37] without. It seems to be loading some websites but not others [23:52:55] weird [23:53:07] bleh; here's why it doesn't work in chrome: http://forums.getfoxyproxy.org/viewtopic.php?f=4&t=629 [23:53:17] because it doesn't send dns requests through the proxy [23:53:56] however, you can use foxyproxy in chrome without url pattern matching [23:54:14] you can select "use labs for all connections" [23:54:36] then when you are done select "Disable FoxyProxy" [23:54:59] I can't get to google.co.uk and I can't find FF add-ons [23:55:42] Ryan_Lane, sorry for slow reply, so a long time until something such as (https://labsconsole.wikimedia.org/wiki/New_Project_Request/proposals) would be able to create an instance under its own 'project' in labs ? [23:55:57] PROBLEM Total processes is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS WARNING: 154 processes [23:56:26] addshore: is this for new wikis? [23:56:36] Disabled a bunch of extensions and suddenly it works [23:56:43] addshore: I'm pretty sure incubator should be used for that [23:56:47] yes, for proposals that are currently open on wikimedia [23:57:21] ahh, but not incubator worthy as incubator is for new language wikis of existing projects, but not for new projects or proposals alltogether [23:57:33] ah [23:57:34] I see [23:58:11] as long as the projects aren't meant to be long-term or production supported [23:58:18] nope [23:59:38] addshore: created and added you as the project admin [23:59:54] In theroy a wiki could be created while a proposal was ongoing, if declined it could be closed, if accepted the content from the wiki could then be moved either straight onto anew project by WM or to incubator [23:59:58] ty :)