[00:06:53] I'm on Wikitech and on the left sidebar I only see "manage service groups". Shouldn't there also be a link for managing instances? [00:07:25] Krenair: ^ I think the user-group based sidebar stuff is missing for a bunch of people [00:08:09] looking [00:09:21] harej, okay so, you see the labs users section but not the labs projectadmins section? [00:09:45] Right [00:10:02] which projects are you an admin of? [00:10:09] Librarybase and Wpx [00:10:18] I'd look it up, but someone broke ldaplist :P [00:12:21] yes you are... huh [00:13:04] harej: btw, in the meantime you can use http://wikitech.wikimedia.org/wiki/Special:NovaInstance directly [00:14:23] and in the instance name, do I need to specify the project name, or is that automatic? [00:15:35] (this is the second time i am doing this, ever) [00:15:50] harej: highly reccomend using something like 'instancepurpose-numeral> [00:15:56] but non-generic ones [00:16:01] db-01 is bad [00:16:05] wpx-db-01 is good [00:16:15] but do I need to write "wpx-" or is that automatic? [00:16:16] remember that instance names are global to all of labs [00:16:18] you have to [00:16:25] okay. [00:16:38] > var_dump( $wgMemc->get( wfMemcKey( 'openstackmanager', 'roles', 'Alex Monk' ) ) ); [00:16:39] array(1) { [00:16:39] [0]=> [00:16:39] string(12) "projectadmin" [00:16:39] } [00:16:47] but you get array(0), weird [00:17:54] I'll try purging your memcached key [00:19:22] Out of curiosity, what would a project use multiple instances for, as opposed to one really big instance? [00:20:25] Hartry now? [00:20:29] harej, try now?* [00:20:36] tabcomplete fail [00:20:53] Krenair: the side bar returns! thank you [00:21:53] harej: multiple instances can be used for splitting up responsibilities for things (eg deployment-prep) or fanning out work/storage over multiple hosts (eg logstash) [00:22:21] or the infamous "grab bag" project that has lots of instances doing only sort of related things [00:22:22] and as failovers [00:22:29] so if one fails others take over [00:22:37] (like tools-checker-01 / -02) [00:22:42] bd808: we still need to move SAL to k8s :D [00:22:59] YuviPanda: yeah, it's still on my workboard [00:23:13] bd808: :D let me know when it hits top :D [00:24:09] "soon"? I've been trying to spend some time offline for the sake of my wrists ;) [00:24:19] anyone else had that dynamic sidebar issue YuviPanda? [00:25:29] bd808: +1 [00:25:34] Krenair: someone many weeks ago [00:25:36] not sure whom [00:25:43] oh, well [00:25:44] it was one of those things someone said when something else was on fire [00:25:46] and I forgot [00:26:50] if they come back or someone else has issues, on silver, `mwscript eval.php labswiki`, $wgMemc->delete( wfMemcKey( 'openstackmanager', 'roles', 'Canonical User Name' ) ); [00:27:29] Krenair: should we file a bug? [00:27:35] bd808: people do that on tool labs. the whole philosophy of having accounts for individuals and then separate accounts for projects that can be re-assigned as necessary; people look at that, go "that's nice," then create a project that's nothing more than a personal hobby horse [00:27:36] probably not worth it [00:27:52] harej: yeah :( [00:27:54] some issue got cached at some point, correct data is now being generated, so... [00:28:05] It would save me a lot of strife if I did that but it appeals to my sense of organization to actually have separate projects. [00:28:07] ok [00:30:29] harej: I haven't succumbed to the grab bag approach on tools except for the bd808-test tool that I piddle with small scripts in before making a real tool account for them. In Labs however I started a horrible "team" project years ago that has a random number of users and instances rotate through it [00:31:04] today I would not recommend making a vaguely scoped project like that [00:32:09] a team project? so, "bd and bd's friends," basically? [00:33:26] harej: mediawiki-core-team. bd and bd's coworkers [00:34:17] which is now "bd and bd's former team mates who are still coworkers but not on the same team" or something [00:34:39] and this is why basing ACLs and such on organisational teams is a bad idea [00:35:40] Yup. and making a Labs project that people drift in and out of makes cleaning things up hard. [00:35:59] Editor-engagement is still around [00:36:13] yeah [00:36:21] lots of others I'm sure too [00:36:25] I've been refusing any 'team' named project recently [00:36:36] some slipped through [00:36:41] * YuviPanda wishes we had written down actual rules [00:36:44] should do that sometime [00:37:41] YuviPanda: WPX is kind of a team based project, but it exists for a specific software project. [00:38:04] I was under the assumption for a long time that making new projects was a bad thing. I no longer suffer under that delusion [00:38:32] harej: yeah. that's cool. A project around a bit of software that people care for is good [00:38:47] a project around a particular group of people is not so good [00:41:05] (03PS60) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [00:46:33] (03CR) 10Ricordisamoa: "PS60 updates grunt-contrib-jshint from ~0.11.3 to ~0.12.0" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [00:48:50] (03PS15) 10Ricordisamoa: Initial commit [labs/tools/faces] - 10https://gerrit.wikimedia.org/r/192096 [00:49:44] (03CR) 10Ricordisamoa: "PS15 updates grunt-contrib-jshint from ~0.11.3 to ~0.12.0" [labs/tools/faces] - 10https://gerrit.wikimedia.org/r/192096 (owner: 10Ricordisamoa) [00:55:14] Also, my project admin thing seems to have disappeared again? [00:56:46] 6Labs, 7Tracking: Create a Labs project for admin tooling enhancements - https://phabricator.wikimedia.org/T123993#1942727 (10lfaraone) 3NEW [00:57:08] 6Labs: Create a Labs project for admin tooling enhancements - https://phabricator.wikimedia.org/T123993#1942727 (10lfaraone) [02:47:35] https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Database#Connecting_to_the_database_replicas_from_other_Labs_instances << is this out of date? iirc labs switch from 192.168 for local IPs to 10.whatever [02:57:58] back [02:58:00] oops [02:58:09] harej, sounds like it might be ancient labs history at this point [02:58:22] all I could find was https://wikitech.wikimedia.org/w/index.php?title=Labs_Eqiad_Migration&diff=next&oldid=98323 and https://wikitech.wikimedia.org/w/index.php?title=Labs_Eqiad_Migration&diff=prev&oldid=99062 [03:03:59] wtf: https://wikitech.wikimedia.org/wiki/Tesla [03:25:08] 6Labs, 10Tool-Labs: Linkwatcher spawns many processes without parent - https://phabricator.wikimedia.org/T123121#1942975 (10Beetstra) @valhallasw: a good solution would be assigning 200-300% processor to the whole task. I found http://wiki.crc.nd.edu/wiki/index.php/Submitting_Batch/SGE_jobs - which suggests "... [04:05:33] 6Labs, 10Tool-Labs, 10DBA, 6Stewards-and-global-tools: Throttling linkwatcher tool user as it is consuming 100% CPU - https://phabricator.wikimedia.org/T121094#1942993 (10Beetstra) Thank you. Not sure if I understand the situation with the privacy, you mean that there is no way to exclude the queries from... [04:06:12] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/A4OWikia Bot was created, changed by A4OWikia Bot link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/A4OWikia_Bot edit summary: Created page with "{{Tools Access Request |Justification=User may access their social media accounts direct from this tool |Completed=false |User Name=A4OWikia Bot }}" [04:18:28] 6Labs, 10DBA, 10wikitech.wikimedia.org: Untangle wikitech/labtestwikitech and s7 DBs and networking and mysql grants - https://phabricator.wikimedia.org/T124002#1943031 (10Krenair) 3NEW a:3Andrew [04:35:09] 6Labs, 10DBA, 10wikitech.wikimedia.org: Untangle wikitech/labtestwikitech and s7 DBs and networking and mysql grants - https://phabricator.wikimedia.org/T124002#1943039 (10Andrew) I think I'm fine with turning it off... it isn't working now anyway, is it? [04:56:37] (03PS1) 10Aude: Add data-values/* to wikidata-feed irc channel [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/264926 [04:57:35] (03CR) 10Aude: [C: 04-1] Add wikidata/* to wikidata-feed (031 comment) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/247831 (owner: 10Addshore) [05:56:14] 6Labs, 10Tool-Labs, 10DBA, 6Stewards-and-global-tools: Throttling linkwatcher tool user as it is consuming 100% CPU - https://phabricator.wikimedia.org/T121094#1943161 (10Beetstra) Grr, I noted a bug on one of the counts (resolved) - it is now counting those and filling the proper table to reduce the count... [08:06:43] andrewbogott: I'm getting 'Puppet is failing to run on the "tools-submit" instance in the Wikimedia Labs [08:06:44] project "tools"' mails every minute now [08:07:52] Yeah, I fixed the ‘every minute’ thing but that would only take effect if puppet ran :) [08:07:54] "(Reason: 'T123186 -- increased debuging for nslcd -- valhallasw 17 jan 2016’);" [08:08:04] shall I re-enable? [08:09:05] Ah, OK. Eh, yes, I should have re enabled it, sorry about that [08:09:51] I won't have SSH access until Wednesday evening [08:10:13] ok, re-enabled, it should hush up shortly [08:10:27] Thanks! [08:10:45] I'm confused why it just started at 9 though [08:11:09] But the message works, so that's good! :) [08:11:15] The cron that sends those emails only runs once per day (or, briefly, every minute during one hour) [08:11:27] and only alerts if puppet has been broken for 24 hours already [08:11:35] Ah, that makes sense [08:11:40] hm, lots of apt complains on that instance [08:11:45] should I care? [08:16:56] valhallasw`cloud: apt is freaking out while trying to install openjdk-7-jre-headless — is that what you were working on before? [08:30:22] 6Labs, 10Tool-Labs: puppet/apt issues on tools-submit - https://phabricator.wikimedia.org/T124014#1943310 (10Andrew) 3NEW [08:43:36] Hoi, there seems to be a problem I can not start autolist .. https://tools.wmflabs.org/autolist/ [08:55:53] 10Tool-Labs-tools-Other, 10Possible-Tech-Projects: Fix TreeViews to provide pageviews statistics for all articles of any wikiproject etc. - https://phabricator.wikimedia.org/T56184#1943343 (10Nemo_bis) [09:00:29] 10Tool-Labs-tools-Other, 10Possible-Tech-Projects: Fix TreeViews to provide pageviews statistics for all articles of any wikiproject etc. - https://phabricator.wikimedia.org/T56184#1943347 (10Nemo_bis) I've clarified the summary so that it's clear what pages we're talking about, what languages (all) and what o... [09:04:38] andrewbogott: no, was trying to debig the cron/nslcd load isaues [09:04:53] GerardM: please file a bug in phab [09:05:31] andrewbogott: there are some more instances with weird apt errors... More things to do :( [09:05:42] phabricator does not work for me [09:06:09] it fails in authentication [09:06:13] it is a muddle [09:10:09] twentyafterfour: ^ [09:10:43] GerardM: when did you last try logging in? That would help twentyafterfour or others with debugging [09:11:16] there is a process defined in Phabricator on restarting tools [09:14:01] ? [09:14:24] Yeah I'll need a lot more info. Are you authenticating with oauth or ldap? [09:39:11] that is where there is a muddle [09:39:22] the one that does not work takes precedence [09:46:46] Huh? In the login window, you choose one or the other [09:47:30] Do you click the mediawiki button? Or do you fill in username and password (wikitech ones)? [10:20:34] 6Labs, 10Tool-Labs: puppet/apt issues on tools-submit - https://phabricator.wikimedia.org/T124014#1943426 (10scfc) a:3scfc Looking into. [10:28:49] 6Labs, 10Tool-Labs: puppet/apt issues on tools-submit - https://phabricator.wikimedia.org/T124014#1943434 (10scfc) It boils down to: ``` scfc@tools-submit:~$ sudo apt-get install libsndfile1 Reading package lists... Done Building dependency tree Reading state information... Done libsndfile1 is already... [10:36:52] 6Labs, 10Tool-Labs: puppet/apt issues on tools-submit - https://phabricator.wikimedia.org/T124014#1943464 (10scfc) a:5scfc>3yuvipanda Note `PreDepends: multiarch-support`: ``` scfc@tools-submit:~$ aptitude show -v libsndfile1 Package: libsndfile1 State: unpacked Automatically installe... [11:16:58] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/A4OWikia Bot was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=261510 edit summary: [11:33:10] 6Labs, 10Tool-Labs, 10DBA: Unthrottle faebot - https://phabricator.wikimedia.org/T123108#1943541 (10Fae) Probably 5 connections is sufficient. Some of my reports rely on more that one database link at the same time to make the SQL, though most are one at a time even when multiple queries are made for the rep... [11:53:52] 10Quarry: Make the table sortable - https://phabricator.wikimedia.org/T71265#1943595 (10He7d3r) [12:03:04] 6Labs, 10DBA, 10wikitech.wikimedia.org: Untangle wikitech/labtestwikitech and s7 DBs and networking and mysql grants - https://phabricator.wikimedia.org/T124002#1943607 (10Krenair) It's working from wikitech (silver) because of some special grants that labtestwikitech (labtestweb2001) doesn't appear to have. [13:01:40] 6Labs, 10Labs-Infrastructure, 10DBA: Inconsistencies between labs replicas - https://phabricator.wikimedia.org/T123985#1943717 (10jcrespo) a:3jcrespo [13:01:48] 6Labs, 10Labs-Infrastructure, 10DBA: Inconsistencies between labs replicas - https://phabricator.wikimedia.org/T123985#1942428 (10jcrespo) p:5Triage>3Normal [13:12:49] 6Labs: Webservice stuck, won't stop, can't restart - https://phabricator.wikimedia.org/T124038#1943742 (10Magnus) 3NEW [13:19:42] 6Labs, 6Stewards-and-global-tools: Create a Labs project for admin tooling enhancements - https://phabricator.wikimedia.org/T123993#1943756 (10JEumerus) Seeing as CU, OS and that functionary stuff is typically associated with that project, I'll add it there as well. [13:21:58] 6Labs, 10Tool-Labs, 10DBA, 6Stewards-and-global-tools: Throttling linkwatcher tool user as it is consuming 100% CPU - https://phabricator.wikimedia.org/T121094#1943758 (10jcrespo) > Not sure if I understand the situation with the privacy, you mean that there is no way to exclude the queries from other peop... [13:32:12] 6Labs, 10Tool-Labs, 10DBA: Unthrottle faebot - https://phabricator.wikimedia.org/T123108#1943783 (10jcrespo) 5Open>3Resolved a:3jcrespo I've granted you 5 concurrent connections. I will be monitoring memory usage per user, if it grows close to crashing a server, I will throttle it again and kill your l... [13:39:02] 6Labs, 10Labs-Infrastructure, 10DBA: Inconsistencies between labs replicas - https://phabricator.wikimedia.org/T123985#1943807 (10jcrespo) [13:40:07] 6Labs, 10Labs-Infrastructure, 10DBA: Inconsistencies between labs replicas - https://phabricator.wikimedia.org/T123985#1942428 (10jcrespo) [13:43:34] 6Labs, 10Labs-Infrastructure, 10DBA: Inconsistencies between labs replicas - https://phabricator.wikimedia.org/T123985#1943812 (10jcrespo) This seems to be an issue with labsdb1003 only, as not only the other hosts are ok, the original production host where filtering happened is ok, too. [14:02:05] 6Labs, 10Tool-Labs: tools.taxonbot and tools.giftbot cronjobs not firing - https://phabricator.wikimedia.org/T123186#1943849 (10doctaxon) Here the next problem again (tools.taxonbot): ``` JSUB_OPTIONS=-once -j y -quiet -v LC_ALL=en_US.UTF-8 -mem 1g 0 */2 * * * jsub -once -j y -quiet -v LC_ALL... [14:04:41] 6Labs, 10Phragile, 6TCB-Team: Unable to access Phragile WMFLabs instance - https://phabricator.wikimedia.org/T123369#1943865 (10Jakob_WMDE) I still have the same problem as well. Accessing this particular instance is not working but any other instance from the same or a different project works fine. [14:08:30] 6Labs, 10DBA, 10wikitech.wikimedia.org: Untangle wikitech/labtestwikitech and s7 DBs and networking and mysql grants - https://phabricator.wikimedia.org/T124002#1943886 (10jcrespo) Very related: https://logstash.wikimedia.org/#dashboard/temp/AVJaNbejptxhN1XaHYzZ [14:18:02] 6Labs: Webservice stuck, won't stop, can't restart - https://phabricator.wikimedia.org/T124038#1943907 (10scfc) All three tools's webservices run on `tools-webgrid-lighttpd-1412` which seems to be overloaded as on login it is hanging after "The last Puppet run was at Mon Jan 18 17:09:23 UTC 2016 (1260 minutes ag... [14:28:32] 6Labs, 10Labs-Infrastructure, 10DBA: Inconsistencies between labs replicas - https://phabricator.wikimedia.org/T123985#1943926 (10jcrespo) ``` root@iron:~$ mysql -h db1069 -P3317 fawiki -e "SELECT count(*) FROM imagelinks" +----------+ | count(*) | +----------+ | 6007034 | +----------+ root@iron:~$ mysql -h... [14:33:02] (03PS17) 10ArthurPSmith: Added a Wikidata-based "chart of the nuclides" under /nuclides [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/245591 [14:33:30] temporarily increase of lag on labs s7 in order to fix some corruption [14:34:04] this is expected and will only take some minutes: https://tools.wmflabs.org/replag/ [14:36:26] (03PS18) 10ArthurPSmith: Added a Wikidata-based "chart of the nuclides" under /nuclides [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/245591 [14:37:19] 6Labs: Webservice stuck, won't stop, can't restart - https://phabricator.wikimedia.org/T124038#1943953 (10scfc) I've rebooted the instance, and looking at the console log on `wikitech`: ``` [1691760.896155] INFO: task jbd2/vda1-8:175 blocked for more than 120 seconds. [1691760.902121] Not tainted 3.13.0-6... [14:38:13] (03CR) 10ArthurPSmith: [C: 031] "Ok, I worked on getting the SPARQL queries to the new wikidata query API working, and it's MUCH faster - very usable. What do you think? T" [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/245591 (owner: 10ArthurPSmith) [14:39:25] 6Labs: Webservice stuck, won't stop, can't restart - https://phabricator.wikimedia.org/T124038#1943961 (10scfc) 5Open>3Resolved a:3scfc All webservices are responding AFAICS now having been restarted by the webservice watcher. I'll leave investigating the underlying issue to T123835. [14:40:45] 6Labs, 10Tool-Labs: tools-webgrid-lighttpd-1204 locked up - https://phabricator.wikimedia.org/T123835#1943966 (10scfc) Same issue today on `tools-webgrid-lighttpd-1412` (T124038): ``` [1691760.896155] INFO: task jbd2/vda1-8:175 blocked for more than 120 seconds. [1691760.902121] Not tainted 3.13.0-62-ge... [14:47:59] 6Labs, 10Labs-Infrastructure, 10DBA: Inconsistencies between labs replicas - https://phabricator.wikimedia.org/T123985#1943979 (10jcrespo) Imagelinks has been reimported: ``` root@iron:~$ mysql -h s7-master fawiki -e "SELECT count(*) FROM imagelinks" +----------+ | count(*) | +----------+ | 6007121 | +----... [14:56:35] 6Labs, 10Labs-Infrastructure, 10DBA: Inconsistencies between labs replicas - https://phabricator.wikimedia.org/T123985#1943984 (10jcrespo) 5Open>3Resolved I have synced labs with production. Replication to labs cannot be 100% reliable due to the filtering performed of private information; additionally if... [15:13:28] 6Labs, 5Patch-For-Review: Replicate data between codfw and eqiad - https://phabricator.wikimedia.org/T85606#1944010 (10faidon) This should be done now, right? [15:18:50] 6Labs, 5Patch-For-Review: Replicate data between codfw and eqiad - https://phabricator.wikimedia.org/T85606#1944024 (10jcrespo) AFAIK, there is no machines for labsdb2 or toolsdb hosts. However, we do not guarantee a reliable user database service for what mostly is scratch data (obviously, production database... [15:26:34] 6Labs, 10Labs-Infrastructure, 10Incident-Labs-NFS-20151216, 6operations, 10ops-eqiad: labstore1002 issues while trying to reboot - https://phabricator.wikimedia.org/T98183#1944037 (10Cmjohnson) The new H800 card has been installed. We should probably schedule a time/day to move to ls1002 [15:47:01] 6Labs, 5Patch-For-Review: Rename labcontrol2001 to labtestweb2001 - https://phabricator.wikimedia.org/T123790#1944091 (10Papaul) [15:47:03] 6Labs, 6operations, 10ops-codfw, 5Patch-For-Review: Update tag and racktables for labcontrol2001: renamed to labtestweb2001 - https://phabricator.wikimedia.org/T123841#1944088 (10Papaul) 5Open>3Resolved a:3Papaul Complete [15:58:47] 6Labs, 10Tool-Labs: webservice restart times out - https://phabricator.wikimedia.org/T124051#1944119 (10russblau) 3NEW [16:17:29] 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#1944221 (10Milimetric) My opinion: We should bring together the good design ideas and details from the graphs everyone finds useful above, and design a mockup o... [16:24:25] 6Labs, 6Stewards-and-global-tools: Create a Labs project for admin tooling enhancements - https://phabricator.wikimedia.org/T123993#1944236 (10Luke081515) But remeber: If you use a proxy (typically to reach a wiki at labs), the IP is always the proxy IP, not the "real" IP of the user, if you check him (so all... [16:24:38] 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#1944237 (10NiharikaKohli) >>! In T120497#1944221, @Milimetric wrote: > My opinion: > > We should bring together the good design ideas and details from the graph... [16:30:26] Go figure, for a moment https://tools.wmflabs.org/catscan2/catscan2.php gave 404 [16:30:34] Maybe a periodic restart was ongoing? [16:37:56] 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#1944280 (10Milimetric) > That sounds like a good idea. Should we get in touch with Design about this? That's ideal, but not necessary. We can come up with some... [16:42:56] 6Labs, 10Tool-Labs: webservice restart times out - https://phabricator.wikimedia.org/T124051#1944298 (10scfc) 5Open>3Resolved a:3scfc I have force-deleted the job with `qdel -f 2479270` and manually killed `lighttpd` on `tools-webgrid-lighttpd-1205`. [17:14:58] 6Labs, 10Tool-Labs: Install pthreads on Tool Labs - https://phabricator.wikimedia.org/T123945#1944379 (10kaldari) [17:15:42] kaldari, ringa ping ping [17:16:09] hello [17:18:07] didn't have any luck with the pthreads request :( [17:24:09] 6Labs, 10Tool-Labs-tools-Other, 6Commons: Provide service to filter over categorization from a list of Commons categories - https://phabricator.wikimedia.org/T110833#1944385 (10Multichill) [17:38:47] 6Labs: beta swift labs instances requirements - https://phabricator.wikimedia.org/T123512#1944421 (10fgiunchedi) @chasemp @yuvipanda @andrew I'm assuming there's enough capacity to bump the limit and how to do that? [18:03:55] anomie, where would the temp directory of a tool be located? [18:04:04] IE where the cookies are stored [18:05:42] Cyberpower678: /tmp, I think, unless you overrode it. [18:07:45] anomie, My cookies aren't there. [18:07:59] And I really want to eat those cookies. :P [18:10:23] Cyberpower678: Try ``php -r 'echo sys_get_temp_dir() . "\n";'`` [18:10:25] ? [18:11:12] It says /tmp [18:11:22] But I don't see any Peachy cookies there. [18:11:35] No idea then. [18:11:42] Cyberpower678: /tmp is going to be on the exec node and not the bastions [18:11:50] because /tmp isn't nfs [18:12:20] bd808, and how do I access that. I want to delete my cookies, otherwise my bots will have a fit with the API [18:12:32] good question... [18:12:37] I just deployed OAuth. [18:12:48] * bd808 looks for the right qstat magic [18:13:30] Cyberpower678: Try just sshing to the exec host from tools-bastion-01? [18:13:55] You mean my cyberbot exec node? [18:13:55] yeah, that's the right thing to do but you need to lookup the right exec host [18:14:11] what tool? [18:14:18] bd808, that's a no brainer. Cyberbot has it's exec node [18:14:22] ah [18:14:22] *its own [18:14:35] hi Earwig, could you take a look at my copyvios PR when you get a chance? [18:14:40] Cyberpower678: well that makes it easier :) [18:14:48] indeed :p [18:15:17] fhocutt: today [18:15:25] Earwig: tyvm! [18:15:31] I suck at linux. [18:15:48] I need a command to list the files of a given directory. [18:16:14] I always forget that one. [18:16:17] ls [18:16:28] ls -al /tmp [18:16:42] OMFG [18:16:57] There's a fuck ton of cookies in there. [18:17:19] (come to the dark side, we have cookies) [18:17:36] bd808, what about a command to delete the entire directory contents? [18:17:51] Without having to type in a delete 250 times. [18:20:13] 6Labs, 10DBA, 10wikitech.wikimedia.org: Untangle wikitech/labtestwikitech and s7 DBs and networking and mysql grants - https://phabricator.wikimedia.org/T124002#1944591 (10Andrew) Alex has pointed out that this dependency means that history pages will break on wikitech if/when the main db servers go down. T... [18:21:00] Cyberpower678: rm *.dat for example [18:21:24] rm * deletes everything? [18:22:10] Cyberpower678: For practical purposes, yes. Technically not, it ignores files whose names begin with '.'. [18:22:18] ^ [18:23:37] There we go. [18:23:41] Thanks everyone [18:32:44] Yay, Cyberbot is now operating using OAuth [18:34:34] anomie, is there any way you can actually verify that as a developer? [18:35:22] Cyberpower678: ... No, I don't think there is. [18:38:47] anomie: Maybe you have an idea: I want to purge pages with math without action=purge&mathpurge=true via index.php. Is there a way to purge this math formulars via api? which parameters do I need? [18:45:44] Luke081515: I have no idea what "mathpurge=true" might do. I don't see anything equivalent in https://en.wikipedia.org/w/api.php?modules=purge and I don't see any actions containing "math" in their name, so whatever the extension does there it doesn't seem to have added the functionality to the API. [18:47:03] anomie: Ok, thanks [18:47:53] tools.wmflabs.org doesn't redirect to www.tools.wmflabs.org. Is this the default behaviour? [18:58:27] rohit-dua: Normally year [18:58:32] *yeah [18:59:21] rohit-dua: In general all domains with a part like thas have no www, for example: "en.wikipedia.org" or "maps.google.com" [19:01:57] Luke081515: I get a "connection was reset" without www part. [19:02:51] 6Labs, 6operations, 7Wikimedia-log-errors: labswiki cannot connect to x1-slave (db1031), and soon, x1-master, either [Error connecting to 10.64.16.20: :real_connect(): (HY000/2003): Can't connect to MySQL server on '10.64.16.20' (4)] - https://phabricator.wikimedia.org/T121866#1944794 (10Andrew) Probably the... [19:03:31] 6Labs, 6operations, 7Wikimedia-log-errors: labswiki cannot connect to x1-slave (db1031), and soon, x1-master, either [Error connecting to 10.64.16.20: :real_connect(): (HY000/2003): Can't connect to MySQL server on '10.64.16.20' (4)] - https://phabricator.wikimedia.org/T121866#1944795 (10Andrew) a:3Andrew [20:50:32] 6Labs, 10DBA, 10wikitech.wikimedia.org, 5Patch-For-Review: Untangle wikitech/labtestwikitech and s7 DBs and networking and mysql grants - https://phabricator.wikimedia.org/T124002#1945566 (10Krenair) So now I think we should work out why silver seems to be able to connect to some s7 slaves and not others,... [20:52:52] 6Labs, 10DBA, 10wikitech.wikimedia.org, 5Patch-For-Review: Untangle wikitech/labtestwikitech and s7 DBs and networking and mysql grants - https://phabricator.wikimedia.org/T124002#1945580 (10jcrespo) It was causing the same kind of problems, but different source: T121866 (left here for reference). [20:53:28] 6Labs, 6operations, 5Patch-For-Review, 7Wikimedia-log-errors: labswiki cannot connect to x1-slave (db1031), and soon, x1-master, either [Error connecting to 10.64.16.20: :real_connect(): (HY000/2003): Can't connect to MySQL server on '10.64.16.20' (4)] - https://phabricator.wikimedia.org/T121866#1945582 (10... [21:00:16] 6Labs, 10DBA, 10wikitech.wikimedia.org, 5Patch-For-Review: Untangle wikitech/labtestwikitech and s7 DBs and networking and mysql grants - https://phabricator.wikimedia.org/T124002#1945610 (10jcrespo) Example error (only db1041 is failing): ``` { "_index": "logstash-2016.01.19", "_type": "mediawiki",... [21:01:20] 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#1945618 (10Nuria) >@Ainali, I think this project might end up being too small for ten students to work on for ten weeks. :) Agreed. This project is well suited f... [21:04:23] anomie, Wikipedia seems to be down. [21:04:49] Cyberpower678: I can load my watchlist fine [21:04:51] Google Chrome could not load the webpage because en.wikipedia.org took too long to respond. The website may be down, or you may be experiencing issues with your Internet connection. [21:05:04] Everything else works just fine [21:05:14] Obviously since I'm on IRC [21:05:36] Wikipedia works for me [21:05:42] Okay, now it's loading again. [21:05:50] It was like that for a good 3 minutes. [21:09:40] 6Labs, 10DBA, 10wikitech.wikimedia.org, 5Patch-For-Review: Untangle wikitech/labtestwikitech and s7 DBs and networking and mysql grants - https://phabricator.wikimedia.org/T124002#1945713 (10jcrespo) I do not think it is grant related- the error would be permission denied, and db1041 is a "generic group" s... [22:47:35] kaldari, ping [23:12:12] 6Labs: labstore2001 disk space WARNING - https://phabricator.wikimedia.org/T123874#1946176 (10yuvipanda) p:5Triage>3High [23:13:13] 6Labs: labstore2001 disk space WARNING - https://phabricator.wikimedia.org/T123874#1939994 (10yuvipanda) `/dev/mapper/backup-tools 8.0T 7.2T 813G 91% /srv/eqiad/tools` So that's an 8T lv and that's almost full. These aren't snapshots but actual live backups, so I don't think we can just drop old backups -... [23:13:26] chasemp: andrewbogott ^ is also stuff that's going to bite us [23:13:32] in the not distant future [23:13:53] I don't see wikibugs atm link? [23:13:57] YuviPanda: https://phabricator.wikimedia.org/T123874 [23:14:14] 6Labs: labstore2001 disk space WARNING - https://phabricator.wikimedia.org/T123874#1946182 (10yuvipanda) on labstore1001, tools only has 6.2T used. I wonder what's the extra space in 2001 used for? [23:14:17] YuviPanda: of course I didn’t do anything but open a case [23:14:17] chasemp: https://phabricator.wikimedia.org/T123874 [23:14:20] bah [23:14:38] um… sorry, I’m slow, you were already editing that bug [23:14:40] andrewbogott: can you ack the icinga alert with a link to the ticket? [23:14:46] one thing to note is [23:14:49] yes [23:14:50] taht is still RAID6 I think [23:14:59] andrewbogott: 1001 or 2001? [23:15:07] err [23:15:09] chasemp: ^ [23:15:48] we think the copy on 2001 is bigger than the actual data on 1001? [23:15:58] yeah [23:15:59] not sure why [23:16:08] 1001 is definitely RAID10, not sure about 2001 [23:16:20] I looked friday pretty sure raid6 [23:16:24] iirc [23:16:29] chasemp: on 2001 you mean? [23:16:31] or 1001? [23:16:33] 2001 [23:16:35] right [23:16:37] that's possible [23:16:50] but would that cause discrepancies [23:16:56] in the size that lvs reports? [23:16:57] err [23:17:00] df reports? [23:17:04] I thought that was at different levels [23:17:19] ok I'm sorry raid 5 I guess [23:17:22] root@labstore2001>cat /proc/mdstat [23:17:22] Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [23:17:24] but [23:17:29] md126 : active (auto-read-only) raid5 sdg[4] sdc[0] sdh[6] sdd[1] sde[2] sdf[3] [23:17:38] YuviPanda: so the solution to the full disk is to delete some files, huh? [23:18:10] YuviPanda: Can you use hhvm on Tool Labs? If so, is there any documentation about that? [23:18:52] kaldari: the hhvm package is installed on trusty nodes [23:19:02] so you can use hhvm following hhvm's docs [23:19:09] instead of 'php ' [23:19:11] you'd do [23:19:13] hhtm [23:19:17] it's the same version of hhvm prod uses [23:19:20] so a bit behind [23:19:44] andrewbogott: possibly. [23:19:58] andrewbogott: it depends. we might have files there that were deleted on labstore1001 but not deleted on 2001 [23:20:02] not sure if the rsync deletes files [23:20:06] if not, we'll have to clean that p [23:20:13] but it's not just snapshots we can drop unfortunately [23:20:16] ooh [23:20:18] it's an option in rsync [23:20:32] Does that mean I can just use it from a regular Tool Labs project account? (I'm sure exactly what "trusty nodes" refers to) [23:20:51] YuviPanda: ^ [23:21:03] I'm not sure exactly... [23:22:00] YuviPanda: the labstore2001 backups are made by storage-replicate? I can add a —delete into the rsync command there [23:22:31] andrewbogott: that might be a fix yeah [23:22:46] kaldari: so by default jsub puts you on precise nodes with older versions of things [23:22:47] kaldari: jsub -l release=trusty program [args...] [23:22:48] if you add [23:22:49] I would imagine [23:22:51] -l release=trusty [23:22:53] hah [23:22:55] what chasemp said [23:22:57] you get newer versions of things [23:23:02] and also things that don't have precise packages [23:23:03] like hhvm [23:23:04] oh, it already has '--delete-during ' [23:23:15] so I think it’s cleaning up already. Do we have evidence that it isn’t? [23:23:37] chasemp: Is there any documentation of this or should I try to create some? [23:23:48] there is https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Grid [23:23:51] not sure how comprehensive [23:23:56] awesome. Thanks [23:24:19] andrewbogott: I think it's just a mystery of why [23:24:22] dev/mapper/backup-tools ext4 8.0T 7.2T 813G 91% /srv/eqiad/tools [23:24:28] is bigger than the labstore1001 equiv [23:24:43] and a lot bigger [23:24:45] dev/mapper/labstore-tools ext4 8.0T 6.2T 1.9T 78% /srv/project/tools [23:24:47] iiuc [23:24:48] maybe —delete-during only modifies —delete? Trying to find documentation one way or another [23:25:10] yeah [23:25:11] one full T bigger seems interesting [23:25:12] there's like [23:25:15] a T of extra stuff [23:25:33] or the same stuff is just bigger w/ striping [23:25:35] on raid5 [23:25:38] and it's a lot of little things [23:25:44] but seems odd if so [23:26:07] totally can’t tell if we need —delete —delete-during or if —delete-during implies —delete [23:28:32] andrewbogott: chasemp can I leave it to you guys to figure out and go get some food? [23:28:50] I don't think it's urgent either way that we know of [23:28:56] sure [23:28:59] it's just a slow burn [23:29:00] I mean, pretty soon we better figure it out but go eat man [23:29:04] yeah [23:29:05] good point [23:29:07] I gotta do the same here [23:29:25] just didn't want it to fall through cracks [23:30:45] gotcha [23:31:09] going away for realz now [23:31:32] YuviPanda: I'm assuming "hhtm" is a typo [23:31:56] andrewbogott: so far it seems the same but more used space so... [23:32:00] open file handles or something odd? [23:32:28] kaldari: yes [23:32:29] hhvm [23:32:33] or bad handling of sparse files? [23:33:55] andrewbogott: is ther ean active backup now? [23:33:58] look at [23:34:02] lsof | grep tool [23:34:16] ah no I tink that's me :D [23:35:11] 6Labs, 10Tool-Labs: (tools.taxonbot) Error: 503, Service Unavailable at Tue, 19 Jan 2016 22:59:53 GMT - https://phabricator.wikimedia.org/T124118#1946237 (10doctaxon) 3NEW a:3yuvipanda [23:36:20] 6Labs, 10Tool-Labs: (tools.taxonbot) Error: 503, Service Unavailable at Tue, 19 Jan 2016 22:59:53 GMT - https://phabricator.wikimedia.org/T124118#1946246 (10yuvipanda) a:5yuvipanda>3None [23:36:32] 6Labs, 10Tool-Labs, 10Wikimedia-General-or-Unknown: (tools.taxonbot) Error: 503, Service Unavailable at Tue, 19 Jan 2016 22:59:53 GMT - https://phabricator.wikimedia.org/T124118#1946237 (10yuvipanda) [23:36:48] 6Labs, 10Tool-Labs, 10Wikimedia-General-or-Unknown: (tools.taxonbot) Error: 503, Service Unavailable at Tue, 19 Jan 2016 22:59:53 GMT - https://phabricator.wikimedia.org/T124118#1946237 (10yuvipanda) This looks like an issue on the mediawiki side perhaps? [23:38:11] 6Labs, 10Tool-Labs, 10Wikimedia-General-or-Unknown: (tools.taxonbot) Error: 503, Service Unavailable at Tue, 19 Jan 2016 22:59:53 GMT - https://phabricator.wikimedia.org/T124118#1946250 (10doctaxon) But the script is running on tools.taxonbot ... [23:42:13] andrewbogott: seems like avail space on the vg if you feel like extending is the best option atm [23:42:14] Free PE / Size 3841840 / 14.66 TiB [23:42:26] but the size discrepancy is pretty intersting [23:42:54] du is taking a long while here [23:43:05] yeah, I just checked a dir that has the same # of files on source and dest but different size on dest [23:43:20] 6Labs, 10Tool-Labs, 10MediaWiki-API, 10Wikimedia-General-or-Unknown: (tools.taxonbot) Error: 503, Service Unavailable at Tue, 19 Jan 2016 22:59:53 GMT - https://phabricator.wikimedia.org/T124118#1946258 (10Giftpflanze) This error originates from the Wikimedia API. [23:43:23] but in this case it’s only barely different [23:43:40] right [23:43:58] but in the case of diff usage it's usually lots of little files that burn you [23:44:02] I think [23:44:22] ok [23:44:28] I don’t immediately know how to resize the volume, do you? [23:45:00] other than that it's lvextend no :) [23:45:13] should be if the VG has spare space [23:45:30] google off [23:47:21] Is it really just lvextend -L+1T /dev/mapper/backup-tools [23:47:26] http://www.tldp.org/HOWTO/LVM-HOWTO/extendlv.html so.... lvextend -L+2000T /dev/backup/tools [23:47:32] :) [23:47:35] whoa, 2000T? [23:47:37] +2000G [23:47:38] sorry [23:47:39] :) [23:47:43] yes, that’s better [23:47:45] I had 2T and then wanted to change it [23:47:46] want to do the honors? [23:47:48] and I'm tired man :) [23:48:16] * andrewbogott expects a 2000Tb thumb drive by 2020 [23:48:22] it's almost 6 here and I have to go in 5, I can to this tomorrow [23:48:25] 6Labs, 10wikitech.wikimedia.org: Decide on future of Semantic extensions on Wikitech - https://phabricator.wikimedia.org/T123599#1946276 (10Reedy) ``` reedy@tin:/srv/mediawiki-staging/php-1.27.0-wmf.11/extensions$ grep -R wfMsg * EducationProgram/RELEASE-NOTES:* Replaced all usages of wfMsg* with their modern... [23:48:26] but I don't want to drop and run [23:48:40] chasemp: I’ll update the bug and then we’ll actually resize at the beginning of a day vs. the end [23:48:44] in case it goes awry [23:48:47] sounds good to me [23:48:53] I think we are ok for the moment [23:48:59] for once we caught something in time [23:49:03] so let's reap the rewards [23:50:01] 6Labs: labstore2001 disk space WARNING - https://phabricator.wikimedia.org/T123874#1946284 (10Andrew) There's no obvious reason for the extra disk usage. File counts don't show additional files on 2001. So, chalking that up to random rsync/block size issues, the solution for now is just to resize the volume.... [23:50:30] sure, andrew, block roundoff is totally responsible for an extra Tb of storage [23:50:32] * andrewbogott shrugs [23:51:00] yeah...I"m running some stuff to see if I have any insight here [23:51:03] but it's super slow to finish [23:53:36] 6Labs, 10Tool-Labs, 10MediaWiki-API, 10Wikimedia-General-or-Unknown: (tools.taxonbot) Error: 503, Service Unavailable at Tue, 19 Jan 2016 22:59:53 GMT - https://phabricator.wikimedia.org/T124118#1946295 (10doctaxon) (tools.taxonbot) Here one more, running the script c-uncat.tcl by grid engine, error repor... [23:58:32] thanks andrewbogott / chasemp! [23:59:20] 6Labs, 10Tool-Labs, 10MediaWiki-API, 10Wikimedia-General-or-Unknown: (tools.taxonbot) Error: 503, Service Unavailable at Tue, 19 Jan 2016 22:59:53 GMT - https://phabricator.wikimedia.org/T124118#1946321 (10bd808) >>! In T124118#1946250, @doctaxon wrote: > But the script is running on tools.taxonbot ... Th...