[01:19:54] 6Labs, 10Tool-Labs: Tool Labs users .bashrc file does not exist for tools accounts - https://phabricator.wikimedia.org/T131561#2170929 (10scfc) The `~/.bashrc` file for users does not come from Puppet, but from the `bash` package: ``` scfc@tools-bastion-05:~$ dpkg-query -S /etc/skel/.bashrc bash: /etc/skel/.... [01:32:38] 6Labs, 10Tool-Labs: Tool Labs users .bashrc file does not exist for tools accounts - https://phabricator.wikimedia.org/T131561#2171200 (10tom29739) It says it's managed by puppet: '#### THIS FILE IS MANAGED BY PUPPET #### as template('base/environment/skel/bashrc')... [01:36:00] 6Labs, 10Tool-Labs: Tool Labs users .bashrc file does not exist for tools accounts - https://phabricator.wikimedia.org/T131561#2171202 (10scfc) @tom29739: I'm sorry, you are right. I meant the mechanism by which the file lands in the home directories and did not check its contents. [01:44:39] 6Labs, 10Tool-Labs: Tool Labs users .bashrc file does not exist for tools accounts - https://phabricator.wikimedia.org/T131561#2171205 (10tom29739) @scfc If the toolwatcher command runs when the tool of created, then why not have it run when the tool account is first logged into instead? Then the PAM rule woul... [03:03:20] horizon reject my login now. Is something there broken? [03:05:13] 2fa is always broken. [03:08:10] YuviPanda: should I use wikitech credentials for horizon or something else? [03:08:57] ah, it now has separate 2FA not the same as phab... [07:35:08] hi ppl [08:09:48] 10Quarry: Quarry stuck on "queued" state - https://phabricator.wikimedia.org/T131586#2171662 (10Edgars2007) [08:16:10] (03CR) 10Jean-Frédéric: [C: 032] Mark Wikipedia tracking page edits as non-minor [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281053 (https://phabricator.wikimedia.org/T110830) (owner: 10Lokal Profil) [08:16:59] (03CR) 10jenkins-bot: [V: 04-1] Mark Wikipedia tracking page edits as non-minor [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281053 (https://phabricator.wikimedia.org/T110830) (owner: 10Lokal Profil) [08:18:28] (03CR) 10Jean-Frédéric: "@Lokal Profil Needs rebasing" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281053 (https://phabricator.wikimedia.org/T110830) (owner: 10Lokal Profil) [08:22:23] (03PS2) 10Lokal Profil: Mark Wikipedia tracking page edits as non-minor [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281053 (https://phabricator.wikimedia.org/T110830) [08:34:53] PROBLEM - Host tools-bastion-01 is DOWN: CRITICAL - Host Unreachable (10.68.17.228) [08:36:30] (03CR) 10Jean-Frédéric: [C: 032] Mark Wikipedia tracking page edits as non-minor [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281053 (https://phabricator.wikimedia.org/T110830) (owner: 10Lokal Profil) [08:42:04] orenbo: you should /join #wmhack too. More chatter happening there from folks here at the hackathon [08:43:52] (03Merged) 10jenkins-bot: Mark Wikipedia tracking page edits as non-minor [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281053 (https://phabricator.wikimedia.org/T110830) (owner: 10Lokal Profil) [09:13:48] (03PS1) 10Lokal Profil: Make address parameter always fussy in api search [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281101 (https://phabricator.wikimedia.org/T41276) [09:21:03] (03CR) 10Jean-Frédéric: [C: 032] Make address parameter always fussy in api search [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281101 (https://phabricator.wikimedia.org/T41276) (owner: 10Lokal Profil) [09:23:20] (03Merged) 10jenkins-bot: Make address parameter always fussy in api search [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281101 (https://phabricator.wikimedia.org/T41276) (owner: 10Lokal Profil) [09:31:23] (03CR) 10Lokal Profil: Add configuration for object monuments in France (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235630 (https://phabricator.wikimedia.org/T111263) (owner: 10Jean-Frédéric) [09:57:01] (03PS2) 10Jean-Frédéric: Add configuration for object monuments in France [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235630 (https://phabricator.wikimedia.org/T111263) [10:03:57] (03PS1) 10Lokal Profil: De-flake monuments_config and add it to tox [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281121 [10:04:34] 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org, 13Patch-For-Review, 5WMF-deploy-2016-03-29_(1.27.0-wmf.19): 2FA seems to be broken on wmf.19 - https://phabricator.wikimedia.org/T131445#2172220 (10Reedy) >>! In T131445#2170869, @dpatrick wrote: > Okay, I'm able to see the error now. Than... [10:36:00] 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org, 13Patch-For-Review, 5WMF-deploy-2016-03-29_(1.27.0-wmf.19): 2FA seems to be broken on wmf.19 - https://phabricator.wikimedia.org/T131445#2172241 (10Reedy) Logged in with my wikitech details, createAndPromote to give me 'crat, and SQL inser... [10:38:28] 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org, 13Patch-For-Review, 5WMF-deploy-2016-03-29_(1.27.0-wmf.19): 2FA seems to be broken on wmf.19 - https://phabricator.wikimedia.org/T131445#2172245 (10Reedy) 5Open>3Resolved a:3Reedy [10:50:27] 6Labs, 10Labs-Infrastructure, 10DBA: Lost database changes on s2 for 3 hours on labs replicas - https://phabricator.wikimedia.org/T129432#2172248 (10Superyetkin) Is there any chance the missing rows in //trwiki// database can be imported? My tools have been experiencing inconsistency issues for a few weeks. [10:58:49] (03CR) 10Jean-Frédéric: [C: 032] De-flake monuments_config and add it to tox [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281121 (owner: 10Lokal Profil) [10:59:24] (03Merged) 10jenkins-bot: De-flake monuments_config and add it to tox [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281121 (owner: 10Lokal Profil) [12:44:42] hmm [12:44:50] how do i add roles on an instance in horizon? [12:47:43] * Reedy has a look [12:48:43] brion: I think we might not have rights to do so [12:48:56] ok i'll go in through wikitech for now :) [12:49:24] Oh, wait [12:49:31] Yeah, no [12:49:41] it should be on the "Images" page [12:49:48] There should be a launch button on the right column [12:49:51] So I suspect it's permissions [12:49:58] https://albertomolina.files.wordpress.com/2013/11/images.png?w=1252&h=888 [12:50:13] security page throws a few errors [12:50:15] "Error: Unable to retrieve security groups." etc [12:51:51] Can't even upload/generate keypairs etc too [12:52:08] brion: I tell a lie [12:52:21] If I go to a project I actually have rights on (ie deployment prep) [12:52:29] There is a launch button on the images page [12:55:54] Reedy: well i can create an instance from an imgae [12:55:59] but i had no opportunity to add roles to it [12:56:17] eg role::labs::mediawiki_vagrant [12:57:25] Is it not a post event? [12:58:37] Reedy: ? [12:59:16] I mean, do it after the inital setup/config of the macjine [13:08:46] (03PS1) 10Youni Verciti: Rev 0.3 More functions & PWB [labs/tools/vocabulary-index] - 10https://gerrit.wikimedia.org/r/281162 [13:27:29] 10Quarry: All queries only just queued - https://phabricator.wikimedia.org/T131610#2172552 (10Dvorapa) [13:27:41] 10Quarry: All queries only just queued - https://phabricator.wikimedia.org/T131610#2172564 (10Dvorapa) p:5Triage>3Unbreak! [13:36:00] 10Quarry: All queries only just queued - https://phabricator.wikimedia.org/T131610#2172552 (10Harej) Possible duplicate of T131586 [13:39:44] (03PS3) 10Jean-Frédéric: Add configuration for object monuments in France [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235630 (https://phabricator.wikimedia.org/T111263) [13:40:00] (03CR) 10jenkins-bot: [V: 04-1] Add configuration for object monuments in France [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235630 (https://phabricator.wikimedia.org/T111263) (owner: 10Jean-Frédéric) [13:44:59] andrewbogott: Currently here? Can you take a look at rcm-2.rcm.eqiad.wmflabs? SSH ist broken, like after the last labs reboot [14:02:22] (03PS4) 10Jean-Frédéric: Add configuration for object monuments in France [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235630 (https://phabricator.wikimedia.org/T111263) [14:02:40] !log quarry killed 100% CPU using process on quarry-runner-02 (ptrace suggested some sort of idle loop). Let's see if that has any effect. [14:02:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL, Master [14:06:31] !log quarry systemctl restart celery-quarry-worker.service hangs. Will now reboot quarry-runner-02. [14:06:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL, Master [14:11:28] horizon is kicking me out with 'unauthorized. try loggign in again' [14:12:35] brion: Typo and all? [14:12:47] James_F: no. [14:13:02] brion: I'd be unsurprised, and it'd be easier to grep for. ;-) [14:13:02] also, "https://wikitech.wikimedia.org/wiki/Help:Proxy" is either out of date, or wrong maybe? [14:13:11] i can't find the web proxies configuration on wikitech anymore [14:13:17] is that moved to horizon only now? [14:13:23] 10Quarry: All queries only just queued - https://phabricator.wikimedia.org/T131610#2172718 (10valhallasw) 5Open>3Resolved p:5Unbreak!>3High a:3valhallasw Restarted runner-02, and queries seem to be running again. [14:13:55] brion: yes [14:15:33] ok that might be https://bugs.launchpad.net/horizon/+bug/1494548 [14:19:06] (03CR) 10Lokal Profil: [C: 031] "I'm imaginary +2'ing this" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235630 (https://phabricator.wikimedia.org/T111263) (owner: 10Jean-Frédéric) [14:19:24] ok now to figure out how the fuck to edit security groups [14:20:32] ok that's back on wikitech :D [14:22:37] (03CR) 10Jean-Frédéric: [C: 032] "+2-ing this as wished by Lokal Profil :)" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235630 (https://phabricator.wikimedia.org/T111263) (owner: 10Jean-Frédéric) [14:24:02] haha i have to rereate my instance to allow port 8080 [14:24:04] THATS AWESOME [14:24:33] * James_F sighs. [14:24:58] (03Merged) 10jenkins-bot: Add configuration for object monuments in France [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235630 (https://phabricator.wikimedia.org/T111263) (owner: 10Jean-Frédéric) [14:30:55] brion: you can add the port to the security group [14:31:03] and oyu can add new security groups in horizon I think [14:31:16] (to an existing instance) [14:59:55] brion: what are you doing when that happens? [15:00:08] andrewbogott: was looking at an instance detail [15:00:16] hm [15:00:20] got the login screen and error message [15:00:23] so, meaning... [15:00:23] stuck in loop forever [15:00:33] eventually removed the 'nextstep=whatever' bit of the url [15:00:35] and then was fine [15:00:36] you clicked on the instance detail link, and it dumped you back there? [15:00:39] yes [15:00:44] does it happen again? [15:00:45] it may have coincided with a session expiration or something [15:00:49] no idea [15:01:01] is it possible that you're not projectadmin in that project? [15:01:26] (horizon has a habit of this: It shows you an option that you're not allowed to perform, and then does that hilarious instance-log-out-and-punish thing. I'm trying to hunt them all down) [15:04:56] (03PS1) 10Jean-Frédéric: Add lat/lon to Palissy as they are expected in any case [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281176 (https://phabricator.wikimedia.org/T111263) [15:04:58] (03PS1) 10Jean-Frédéric: Add some ignore to .gitignore [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281177 [15:05:00] (03PS1) 10Jean-Frédéric: Add deploy script to toollabs [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281178 [15:05:11] Luke|away: any idea why that instance would explicitly disallow all login usernames, including 'root'? [15:08:04] brion: anyway, if you have an opportunity to reproduce and file a bug, I'm interested. [15:08:11] andrewbogott: i'll try later yeah :D [15:08:22] with luck it's just the same bug as was reported in which case the fix will eventually show up ;) [15:12:44] (03CR) 10Lokal Profil: [C: 031] "+2'ing in my heart" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281177 (owner: 10Jean-Frédéric) [15:14:08] (03CR) 10Lokal Profil: [C: 031] "Looks fine" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281178 (owner: 10Jean-Frédéric) [15:15:07] (03CR) 10Lokal Profil: [C: 031] "Go, go, go!" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281176 (https://phabricator.wikimedia.org/T111263) (owner: 10Jean-Frédéric) [15:15:47] (03CR) 10Jean-Frédéric: [C: 032] Add lat/lon to Palissy as they are expected in any case [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281176 (https://phabricator.wikimedia.org/T111263) (owner: 10Jean-Frédéric) [15:15:53] (03CR) 10Jean-Frédéric: [C: 032] Add some ignore to .gitignore [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281177 (owner: 10Jean-Frédéric) [15:16:05] (03CR) 10Jean-Frédéric: [C: 032] Add deploy script to toollabs [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281178 (owner: 10Jean-Frédéric) [15:16:20] (03Merged) 10jenkins-bot: Add lat/lon to Palissy as they are expected in any case [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281176 (https://phabricator.wikimedia.org/T111263) (owner: 10Jean-Frédéric) [15:16:35] (03Merged) 10jenkins-bot: Add some ignore to .gitignore [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281177 (owner: 10Jean-Frédéric) [15:17:00] (03Merged) 10jenkins-bot: Add deploy script to toollabs [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281178 (owner: 10Jean-Frédéric) [15:45:40] 6Labs, 10MediaWiki-extensions-OATHAuth, 10Wikimedia-Hackathon-2016, 10wikitech.wikimedia.org, and 2 others: 2FA seems to be broken on wmf.19 - https://phabricator.wikimedia.org/T131445#2172967 (10bd808) [15:54:30] anyone have pandas package installed locally in lab so I can add it to my PYTHONPATH? It takes long to install it with git clone :( [16:34:34] Can indexes be created for user-space databases on tool labs? [16:34:50] I tried doing it just now and I got denied, but I may be doing something wrong [16:41:55] hey folks. I can't seem to find the "Manage web proxies" link. Where did it go? [16:42:31] harej: You'd hope so [16:42:57] Reedy: let me know if I'm egregiously messing up here: [16:42:59] MariaDB [s52475__wpx_p]> create index projectindex_pageindex on projectindex (pi_page); [16:43:00] ERROR 1142 (42000): INDEX command denied to user 'u10090'@'10.68.23.74' for table 'projectindex' [16:43:53] That looks vaguely right [16:44:57] s52475__wpx_p is the right db name? [16:45:03] yes [16:46:10] should horizon.wikimedia.org work now? I'm unable to login [16:46:10] Is there anything in the table? [16:46:55] There's lots of stuff in the table. [16:47:18] Over 11 million rows by my count. [16:47:24] I was wondering if dropping it and creating it with indexes would work [16:47:45] So, interestingly, the table is dropped every 24 hours or so when I re-generate it [16:48:17] Meaning we could test this and it would not be all that disruptive [16:48:34] Or we just create a new table :) [16:48:38] Sure. [16:48:46] I'm just testing it on my userdb [16:50:33] halfak, did you see labs-l? [16:50:38] tgr, yes [16:50:42] MariaDB [u1226__test]> CREATE TABLE page ( page_id int unsigned NOT NULL PRIMARY KEY AUTO_INCREMENT, page_title varchar(255) binary NOT NULL ) ENGINE=`innodb`; [16:50:43] Query OK, 0 rows affected (0.11 sec) [16:50:43] MariaDB [u1226__test]> CREATE INDEX title ON page(page_title); [16:50:51] Query OK, 0 rows affected (0.13 sec) [16:50:51] Records: 0 Duplicates: 0 Warnings: 0 [16:52:27] harej: Um [16:52:47] harej: Are you logged into mysql with your user creds? Or the tools creds? [16:53:01] Ooooh, good question [16:53:11] I probably forgot to do "become whatever" [16:53:14] yeah [16:53:23] probably your user can read, maybe write [16:53:30] altering tables less likely [16:54:00] So, if I wanted to create an index on pi_page, would I do: create index projectindex_pageindex on projectindex (pi_page); [16:55:02] that should work fine [16:56:43] and I drop the projectindex table every 24 hours; do I need to re-create the index every time? or is the index a standing view? [16:57:35] no, you'd need to create it [16:57:48] you can inline the statement into the create table, so it's one operation [16:57:49] 6Labs, 10Horizon: Unable to login on Horizon - https://phabricator.wikimedia.org/T131630#2173147 (10Tgr) [16:58:05] Though, depending on the amount of data, it's generally quicker to index after adding data, rather than before [16:59:03] 6Labs, 10Horizon: Unable to login on Horizon - https://phabricator.wikimedia.org/T131630#2173163 (10Reedy) Clock drift on your phone? If so, it's a known issue, with a fix proposed, not sure if it was deployed yet as @chasemp was having the same issue yesterday [17:00:13] 6Labs, 10Horizon: Unable to login on Horizon - https://phabricator.wikimedia.org/T131630#2173165 (10Krenair) I've also heard of @chasemp (Rush) and @SMalyshev (Smalyshev) having difficulty getting in [17:00:47] 6Labs, 10Horizon: Unable to login on Horizon - https://phabricator.wikimedia.org/T131630#2173169 (10Reedy) https://gerrit.wikimedia.org/r/#/c/281033/ was the fix It was merged, so I presume it's deployed... [17:02:13] tgr: Is your clock on your phone in sync? [17:29:50] 6Labs, 10Horizon: Unable to login on Horizon - https://phabricator.wikimedia.org/T131630#2173185 (10Tgr) >>! In T131630#2173163, @Reedy wrote: > Clock drift on your phone? It's within 5 sec of time.is. [17:33:21] (03PS1) 10Lokal Profil: Removing unused flickrripper from repo [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281212 [18:06:36] (03PS1) 10Lokal Profil: Replace be-x-old by be-tarask throughout [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281220 (https://phabricator.wikimedia.org/T112415) [19:24:46] andrewbogott: I only rebootet that instance. After the last reboot you said, that the ssh service or somerhing else responsible for that didn't start IIRC [19:26:01] RECOVERY - Puppet staleness on tools-bastion-05 is OK: OK: Less than 1.00% above the threshold [3600.0] [19:52:35] (03CR) 10Jean-Frédéric: "check experimental" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281220 (https://phabricator.wikimedia.org/T112415) (owner: 10Lokal Profil) [19:59:02] (03CR) 10Jean-Frédéric: "Check experimental w" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281212 (owner: 10Lokal Profil) [20:07:43] andrewbogott: Ok, my login is posssible again, after a reboot at wikitech [20:07:54] huh. OK :) [20:14:31] andrewbogott: I'm currently trying to reproduce this, if this is a general issue [20:18:40] andrewbogott: Ok, seems like this is not a general issue, was jsut a problem of my singel instance, I can't reproduce it [23:10:22] PROBLEM - Puppet run on tools-exec-1210 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [23:12:28] PROBLEM - Puppet run on tools-webgrid-lighttpd-1403 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [23:22:43] PROBLEM - Puppet run on tools-webgrid-lighttpd-1201 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [23:22:43] PROBLEM - Puppet run on tools-exec-1209 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [23:27:49] PROBLEM - Puppet run on tools-webgrid-lighttpd-1203 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [23:28:19] PROBLEM - Puppet run on tools-webgrid-lighttpd-1404 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [23:32:27] !log rcm.cac updating the repos [23:32:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm.cac/SAL, Master [23:40:20] PROBLEM - Puppet run on tools-worker-1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [23:45:20] RECOVERY - Puppet run on tools-exec-1210 is OK: OK: Less than 1.00% above the threshold [0.0] [23:51:25] PROBLEM - Puppet run on tools-webgrid-lighttpd-1202 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [23:52:33] RECOVERY - Puppet run on tools-webgrid-lighttpd-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [23:57:51] RECOVERY - Puppet run on tools-exec-1209 is OK: OK: Less than 1.00% above the threshold [0.0]