[07:13:43] andrewbogott_afk: if you need to mute someone here you can do this [07:13:47] @q shinken-wm [07:13:52] !unq shinken-wm [07:14:02] hmm [07:14:07] @unq shinken-wm [07:14:10] this :P [09:54:28] (03PS1) 10Sitic: Upgrade to AngularJS 1.4, added angular-translate-storage [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/214586 [09:54:46] (03CR) 10Sitic: [C: 032 V: 032] Upgrade to AngularJS 1.4, added angular-translate-storage [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/214586 (owner: 10Sitic) [10:05:16] 10Tool-Labs-tools-Other: ForrestBot should run for WMF-deployed extensions as well as MediaWiki core - https://phabricator.wikimedia.org/T100347#1320135 (10valhallasw) [11:23:21] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1401 is CRITICAL 55.56% of data above the critical threshold [0.0] [11:48:20] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1401 is OK Less than 1.00% above the threshold [0.0] [12:27:05] !log tools created two redis instances (tools-redis-01 and tools-redis-02), beginning to set up stuff [12:27:08] Logged the message, Master [12:41:32] 10Tool-Labs, 5Patch-For-Review, 3ToolLabs-Goals-Q4: Make tools redis hot switchover-able - https://phabricator.wikimedia.org/T99737#1320535 (10yuvipanda) Doing this now. [12:46:20] PROBLEM - Puppet failure on tools-exec-1211 is CRITICAL 20.00% of data above the critical threshold [0.0] [12:47:08] PROBLEM - Puppet failure on tools-exec-1403 is CRITICAL 33.33% of data above the critical threshold [0.0] [12:47:30] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1207 is CRITICAL 12.50% of data above the critical threshold [0.0] [12:47:50] PROBLEM - Puppet failure on tools-webgrid-generic-1403 is CRITICAL 30.00% of data above the critical threshold [0.0] [12:47:52] PROBLEM - Puppet failure on tools-exec-1406 is CRITICAL 40.00% of data above the critical threshold [0.0] [12:48:28] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1209 is CRITICAL 40.00% of data above the critical threshold [0.0] [12:49:42] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1406 is CRITICAL 50.00% of data above the critical threshold [0.0] [12:49:52] PROBLEM - Puppet failure on tools-exec-1405 is CRITICAL 50.00% of data above the critical threshold [0.0] [12:50:14] PROBLEM - Puppet failure on tools-exec-1218 is CRITICAL 66.67% of data above the critical threshold [0.0] [12:50:18] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1202 is CRITICAL 55.56% of data above the critical threshold [0.0] [12:51:01] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1408 is CRITICAL 50.00% of data above the critical threshold [0.0] [12:51:05] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1410 is CRITICAL 55.56% of data above the critical threshold [0.0] [12:51:12] 'tis ok, shinken-wm [12:51:23] !log disable puppet on all toollabs hosts for tools-redis update [12:51:24] disable is not a valid project. [12:51:27] !log tools disable puppet on all toollabs hosts for tools-redis update [12:51:27] is not a valid project. [12:52:28] !log tools disable puppet on all toollabs hosts for tools-redis update [12:52:31] Logged the message, Master [12:59:13] PROBLEM - Host tools-redis-02 is DOWN: CRITICAL - Host Unreachable (10.68.18.69) [13:01:06] PROBLEM - Host tools-redis-01 is DOWN: CRITICAL - Host Unreachable (10.68.18.65) [13:01:42] !log tools recreating tools-redis-01 and -02 [13:01:44] Logged the message, Master [13:03:30] PROBLEM - Puppet failure on tools-webproxy-02 is CRITICAL 20.00% of data above the critical threshold [0.0] [13:16:05] RECOVERY - Host tools-redis-01 is UPING OK - Packet loss = 0%, RTA = 0.97 ms [13:16:43] RECOVERY - Host tools-redis-02 is UPING OK - Packet loss = 0%, RTA = 0.65 ms [13:19:11] PROBLEM - Puppet failure on tools-redis-02 is CRITICAL 11.11% of data above the critical threshold [0.0] [13:24:10] RECOVERY - Puppet failure on tools-redis-02 is OK Less than 1.00% above the threshold [0.0] [13:33:33] RECOVERY - Puppet failure on tools-webproxy-02 is OK Less than 1.00% above the threshold [0.0] [13:35:58] !log tools enable puppet on all hosts, redis move-around completed [13:36:01] Logged the message, Master [13:37:37] sitic: hey! there's going to be a hiccup in tools-redis for a few seconds, just a fyi [13:39:15] !log tools tools-redis-01 is redis master now [13:39:18] Logged the message, Master [13:41:46] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1205 is CRITICAL 20.00% of data above the critical threshold [0.0] [13:42:27] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1201 is CRITICAL 22.22% of data above the critical threshold [0.0] [13:43:14] PROBLEM - Puppet failure on tools-webgrid-generic-1404 is CRITICAL 22.22% of data above the critical threshold [0.0] [13:43:38] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1407 is CRITICAL 40.00% of data above the critical threshold [0.0] [13:43:44] PROBLEM - Puppet failure on tools-webgrid-generic-1402 is CRITICAL 20.00% of data above the critical threshold [0.0] [13:43:48] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1203 is CRITICAL 20.00% of data above the critical threshold [0.0] [13:44:10] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1208 is CRITICAL 22.22% of data above the critical threshold [0.0] [13:44:34] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1206 is CRITICAL 20.00% of data above the critical threshold [0.0] [13:44:52] PROBLEM - Puppet failure on tools-webgrid-generic-1401 is CRITICAL 30.00% of data above the critical threshold [0.0] [13:49:23] PROBLEM - Puppet failure on tools-redis is CRITICAL 40.00% of data above the critical threshold [0.0] [13:49:37] PROBLEM - Puppet failure on tools-redis-slave is CRITICAL 60.00% of data above the critical threshold [0.0] [13:51:45] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1205 is OK Less than 1.00% above the threshold [0.0] [13:52:27] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1201 is OK Less than 1.00% above the threshold [0.0] [13:52:35] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1207 is OK Less than 1.00% above the threshold [0.0] [13:52:49] RECOVERY - Puppet failure on tools-webgrid-generic-1403 is OK Less than 1.00% above the threshold [0.0] [13:53:13] RECOVERY - Puppet failure on tools-webgrid-generic-1404 is OK Less than 1.00% above the threshold [0.0] [13:53:29] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1209 is OK Less than 1.00% above the threshold [0.0] [13:53:39] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1407 is OK Less than 1.00% above the threshold [0.0] [13:53:47] RECOVERY - Puppet failure on tools-webgrid-generic-1402 is OK Less than 1.00% above the threshold [0.0] [13:53:49] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1203 is OK Less than 1.00% above the threshold [0.0] [13:54:11] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1208 is OK Less than 1.00% above the threshold [0.0] [13:54:33] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1206 is OK Less than 1.00% above the threshold [0.0] [13:54:43] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1406 is OK Less than 1.00% above the threshold [0.0] [13:54:53] RECOVERY - Puppet failure on tools-webgrid-generic-1401 is OK Less than 1.00% above the threshold [0.0] [13:55:19] RECOVERY - Puppet failure on tools-exec-1218 is OK Less than 1.00% above the threshold [0.0] [13:55:19] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1202 is OK Less than 1.00% above the threshold [0.0] [13:55:56] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1408 is OK Less than 1.00% above the threshold [0.0] [13:56:02] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1410 is OK Less than 1.00% above the threshold [0.0] [13:56:18] RECOVERY - Puppet failure on tools-exec-1211 is OK Less than 1.00% above the threshold [0.0] [13:57:06] RECOVERY - Puppet failure on tools-exec-1403 is OK Less than 1.00% above the threshold [0.0] [13:57:52] RECOVERY - Puppet failure on tools-exec-1406 is OK Less than 1.00% above the threshold [0.0] [13:59:53] RECOVERY - Puppet failure on tools-exec-1405 is OK Less than 1.00% above the threshold [0.0] [14:03:27] it's possible to ssh from bastion to webgrid node but not to exec node "Permission denied (publickey).", it's intentional? [14:04:29] hey phe [14:04:36] it isn't. which exec node? [14:07:46] !log deployment-prep upgrade java on deployment-restbase0[12] to the 7u79 security update [14:07:49] Logged the message, Master [14:08:09] tools-exec-1207 tools-exec-1204 [14:08:29] all exec nodes I tried apparently [14:09:37] phe: needs to be manually enabled apparently :( [14:09:51] phe: I can enable it on a particular one if you're looking for it right now, and work on a general solution for all shortly [14:10:21] YuviPanda, 1207 please [14:10:26] moment [14:12:11] phe: oh, it's enabled already on 1207 [14:12:12] * YuviPanda checks [14:12:32] aaah [14:12:33] no [14:12:34] fixing [14:14:14] phe: try now [14:14:26] works now, ty [14:36:03] yay better redis reliability [14:36:13] YuviPanda: wikibugs needs a kick though [14:37:16] these redis pubsub libraries don't seem to take it very well if redis is unavailable for a short time, always needs a restart :-/ [16:03:18] where art thou wikibugs? [16:10:47] Wherefore? [16:58:51] 10Tool-Labs, 5Patch-For-Review, 3ToolLabs-Goals-Q4: Make tools redis hot switchover-able - https://phabricator.wikimedia.org/T99737#1321242 (10yuvipanda) All done now, needs documentation. [17:03:07] 10Tool-Labs, 5Patch-For-Review, 3ToolLabs-Goals-Q4: Make tools redis hot switchover-able - https://phabricator.wikimedia.org/T99737#1321248 (10yuvipanda) Documented at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Admin [17:06:48] PROBLEM - Host tools-redis-slave is DOWN: CRITICAL - Host Unreachable (10.68.17.150) [18:38:24] 6Labs, 6operations, 7database: Santitize recent wikis: wikimania 2016 and cn.wikimedia.org at labs dbs - https://phabricator.wikimedia.org/T100441#1321522 (10Krenair) [18:38:41] 6Labs: wikimania2016wiki not recognised by sql command - https://phabricator.wikimedia.org/T96638#1321532 (10Krenair) Looks like it doesn't actually work yet, T100441 will probably fix it [18:39:34] 6Labs: wikimania2016wiki not recognised by sql command - https://phabricator.wikimedia.org/T96638#1321540 (10Krenair) Although I can't find wikimania2016wiki in /etc/hosts on tools-bastion-01 [18:49:35] 6Labs, 10Wikimedia-Labs-General, 6operations, 7Tracking, 7database: (Tracking) Database replication services - https://phabricator.wikimedia.org/T50930#1321591 (10chasemp) [18:52:28] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/KamranMackey was created, changed by KamranMackey link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/KamranMackey edit summary: Created page with "{{Tools Access Request |Justification=I plan on using it for creating tools that are useful to people who use Wikimedia's projects such as Wikipedia, Wikidata and other projec..." [18:53:28] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/KamranMackey was modified, changed by KamranMackey link https://wikitech.wikimedia.org/w/index.php?diff=161070 edit summary: [19:50:28] 6Labs, 10Beta-Cluster, 10Labs-Infrastructure, 6operations, 6WMF-NDA: On labs, add warning text + labs Term of Uses link to the motd files - https://phabricator.wikimedia.org/T100837#1321787 (10hashar) 3NEW [20:08:38] 6Labs, 10Beta-Cluster, 10Labs-Infrastructure, 6operations, 6WMF-NDA: On labs, add warning text + labs Term of Uses link to the motd files - https://phabricator.wikimedia.org/T100837#1321833 (10BBlack) Note for tool labs, there's already some motd stuff in puppet like this: https://github.com/wikimedia/o... [20:21:06] 6Labs, 6operations, 7database: Santitize recent wikis: wikimania 2016 and cn.wikimedia.org at labs dbs - https://phabricator.wikimedia.org/T100441#1321864 (10scfc) @jcrespo: I don't know if you mean that, but the views in `enwiki_p` & Co. visible from Labs are maintained with `maintain-replicas/maintain-repl... [20:22:08] how do I make my lighttpd webservice pass a Access-Control-Allow-Origin: * header? [20:23:03] found it [20:23:51] !cors [20:24:10] !toolscors is https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web#Example_configurations [20:24:11] Key was added [20:24:44] > Restarting..............................Timeout: could not stop job in 30s [20:24:46] * legoktm eyes YuviPanda [20:31:37] 6Labs: wikimania2016wiki not recognised by sql command - https://phabricator.wikimedia.org/T96638#1321946 (10scfc) 5Resolved>3Open *argl* I had forgotten that we puppetized `/etc/hosts` completely, so my "fix" only lasted 30 minutes. When I reorganized the puppetization in https://gerrit.wikimedia.org/r/#/... [20:41:34] legoktm: which one? [20:41:55] YuviPanda: 'legobot', it looks like the webservice started and bigbrother restarted it. [20:42:52] 10Tool-Labs: Trusty instances do not show the motd banners - https://phabricator.wikimedia.org/T85307#1321968 (10scfc) [20:42:53] 6Labs, 10Beta-Cluster, 10Labs-Infrastructure, 6operations, 6WMF-NDA: On labs, add warning text + labs Term of Uses link to the motd files - https://phabricator.wikimedia.org/T100837#1321967 (10scfc) [20:43:52] 6Labs, 10Beta-Cluster, 10Labs-Infrastructure, 6operations, 6WMF-NDA: On labs, add warning text + labs Term of Uses link to the motd files - https://phabricator.wikimedia.org/T100837#1321974 (10yuvipanda) Title should probably refer to beta cluster. [20:43:57] legoktm: looking [21:02:34] legoktm: sorry, I lied, wasn't actually looking :P [21:02:50] looks like it's ok now [21:02:53] should get rid of it soon [21:02:57] let me submit a patch [21:59:49] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/KamranMackey was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=161081 edit summary: [22:29:05] 6Labs, 6operations, 10wikitech.wikimedia.org: Rename specific account in LDAP, Wikitech, Gerrit and Phabricator - https://phabricator.wikimedia.org/T85913#1322290 (10Krenair)