[00:40:45] PROBLEM - Puppet run on tools-puppetmaster-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [03:39:14] Nemo_bis: Hi! do you maintain the dumps labs project? (I had to kill a python user script being run on dumps-1 by user hydriz since the load on our nfs servers was spiking due to high write volume) [06:03:34] 06Labs, 15User-Hydriz: Dumps instances occasionally hammer NFS for temporary storage - https://phabricator.wikimedia.org/T134148#2755721 (10Hydriz) This issue has happened again today (and I [[https://wikitech.wikimedia.org/w/index.php?title=User_talk:Hydriz&curid=4319&diff=940156&oldid=162958|was notified]] a... [06:04:25] 06Labs, 15User-Hydriz: Dumps instances occasionally hammer NFS for temporary storage - https://phabricator.wikimedia.org/T134148#2755723 (10Hydriz) p:05Normal>03High Changing task priority, feel free to revert if you disagree. [06:52:21] PROBLEM - Puppet run on tools-webgrid-lighttpd-1411 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [07:32:17] RECOVERY - Puppet run on tools-webgrid-lighttpd-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [07:38:40] RECOVERY - Host tools-secgroup-test-102 is UP: PING OK - Packet loss = 0%, RTA = 0.69 ms [07:47:36] PROBLEM - Host tools-secgroup-test-102 is DOWN: PING CRITICAL - Packet loss = 100% [08:25:40] 06Labs, 15User-Hydriz: Dumps instances occasionally hammer NFS for temporary storage - https://phabricator.wikimedia.org/T134148#2755811 (10Nemo_bis) >>! In T134148#2348239, @Nemo_bis wrote: > In my understanding, the Labs physical hosts still have plenty of free disk Seems all the more true: {F4679449} > I... [10:43:08] * hare waits for everyone to wake up [11:14:57] RECOVERY - Host tools-secgroup-test-103 is UP: PING OK - Packet loss = 0%, RTA = 2.49 ms [11:27:40] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [11:30:19] 10Tool-Labs-tools-stewardbots, 13Patch-For-Review: StewardBot not logged into irc - https://phabricator.wikimedia.org/T149265#2756076 (10MarcoAurelio) Maybe we can think about SASL (SSL of course) instead? [11:38:41] (03PS1) 10Lokal Profil: Make return values from get_new_categories uniform [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/318907 (https://phabricator.wikimedia.org/T149258) [11:42:31] (03CR) 10Jean-Frédéric: [C: 032] Make return values from get_new_categories uniform [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/318907 (https://phabricator.wikimedia.org/T149258) (owner: 10Lokal Profil) [11:43:34] (03Merged) 10jenkins-bot: Make return values from get_new_categories uniform [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/318907 (https://phabricator.wikimedia.org/T149258) (owner: 10Lokal Profil) [11:44:22] (03PS1) 10Tobias Gritschacher: 2ColConflict has been renamed to TwoColConflict [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318908 [11:46:12] RECOVERY - Host secgroup-lag-102 is UP: PING OK - Packet loss = 0%, RTA = 1.55 ms [11:50:04] (03CR) 10WMDE-Fisch: [C: 032] 2ColConflict has been renamed to TwoColConflict [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318908 (owner: 10Tobias Gritschacher) [11:50:41] (03Merged) 10jenkins-bot: 2ColConflict has been renamed to TwoColConflict [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318908 (owner: 10Tobias Gritschacher) [12:01:59] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [12:19:12] once someone wakes up: what is the best practice for allowing a UWSGI app to use your SQL credentials? [12:19:32] even though in my python app i have the .my.cnf file where it needs to be, it somehow isn't still reading it, likely because of the strict permissions [12:20:26] but I'm thinking I should not be doing that, that I should instead have a read-only account set up. But I don't know how to arrange that. [12:24:43] hmm nevermind, wasn't a file permission thing, was a relative dir vs. absolute dir thing [12:26:43] Hare it shouldnt NEED one if i remember right [12:39:46] 10Tool-Labs-tools-Other: create tool to crunch metrics for views (play started) of video and audio files - https://phabricator.wikimedia.org/T116363#2756225 (10harej-NIOSH) We have an API: https://tools.wmflabs.org/mediaplaycounts/api/1/FilePlaycount/date/Donning_PPE-_Engage_Trained_Observer_CDC02.webm/20150101... [14:09:17] 06Labs, 06Operations, 07Tracking: Migrate tools to secondary labstore HA cluster (Scheduled on 11/2) [tracking] - https://phabricator.wikimedia.org/T146154#2756506 (10chasemp) We may need to reschedule as {T149567} is a an issue [14:25:19] 06Labs, 10Labs-Infrastructure: Labs hosts: make reboot checklist - https://phabricator.wikimedia.org/T149569#2756616 (10Andrew) [14:25:29] chasemp: ^ [14:26:18] andrewbogott: is that something you can make traction on this week/ [14:26:20] ? [14:26:26] sure [14:26:29] thanks man [14:26:40] I'm rebooting labvirts tomorrow so, ideally, before then :) [14:26:43] 10Tool-Labs-tools-Other, 06Community-Tech-Tool-Labs, 07Epic: Convert all Labs tools to use cdnjs for static libraries and fonts - https://phabricator.wikimedia.org/T103934#2756633 (10Jdforrester-WMF) [14:26:54] all 14? [14:27:18] does this come w/ new kernel or is 1014 going to be an outlier still? [14:28:32] 1002-1013 [14:28:38] new kernel for hosts and for instances [14:28:56] https://phabricator.wikimedia.org/T148767 [14:45:08] 06Labs, 06Operations: cronspam from labstores, labcontrol, labstestservices - https://phabricator.wikimedia.org/T149574#2756726 (10faidon) [14:45:16] 06Labs, 06Operations: Kill the labtest $realm - https://phabricator.wikimedia.org/T148717#2756741 (10faidon) Ping! [14:46:14] PROBLEM - Host tools-exec-cyberbot is DOWN: CRITICAL - Host Unreachable (10.68.16.39) [14:48:06] ^ that host has been down for months; can it be deleted? [14:50:39] andrewbogott: there is a task were I asked for confirmation on that and crickets but [14:50:40] I think yes [14:50:45] it's been shutdown for a long while [14:51:37] Ah, that's https://phabricator.wikimedia.org/T147805 right? [14:52:10] since in theory it's just an exec node with a special flag… I'm going to delete it [14:53:05] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: Migrate labsdb1005/1006/1007 to jessie - https://phabricator.wikimedia.org/T123731#2756785 (10chasemp) We need to schedule a downtime to do this move from labsdb1005 to labsdb1004. This should be a very short window of actual outage.... [14:53:38] yes [14:55:01] 06Labs, 10Tool-Labs: tools-exec-cyberbot in SHUTOFF state - https://phabricator.wikimedia.org/T147805#2756790 (10Andrew) 05Open>03Resolved a:03Andrew I deleted the instance. [15:08:27] 06Labs: Request creation of community-labs-monitoring labs project - https://phabricator.wikimedia.org/T148569#2756864 (10Andrew) 05Open>03Resolved a:03Andrew Sorry for the delay! This is done now. @Matthewrbowker, you are set as a projectadmin in the new project; you can add new users or projectadmins o... [15:08:29] 06Labs, 07Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#2756867 (10Andrew) [15:17:48] 06Labs, 10Labs-Infrastructure, 10DBA: Move dbproxy1010 and dbproxy1011 to labs-support network, rename them to labsdbproxy1001 and labsdbproxy1002 - https://phabricator.wikimedia.org/T149170#2756903 (10jcrespo) @chasemp - as we talked on the last meeting we need to sort out some architecture decisions with t... [15:24:26] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Create maintain-views user for labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T148560#2756912 (10jcrespo) a:03jcrespo So, 'maintainviews' will be the user used to create the view (you will connect to mysql using that user). viewmaster wi... [15:35:11] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: Migrate labsdb1005/1006/1007 to jessie - https://phabricator.wikimedia.org/T123731#1936600 (10yuvipanda) If we settle on a date and announce on labs-announce... [15:36:38] 06Labs, 07Tracking: Existing Labs project quota increase requests (Tracking) - https://phabricator.wikimedia.org/T140904#2757009 (10Andrew) [15:36:40] 06Labs, 06Services (watching): Request increased quota for services labs project - https://phabricator.wikimedia.org/T148788#2757007 (10Andrew) 05Open>03declined Closing this as it's best fixed properly [15:54:19] 10Tool-Labs-tools-Xtools: Bugs section on articleinfo returns incorrect results - https://phabricator.wikimedia.org/T148046#2757110 (10Matthewrbowker) p:05Triage>03Normal [16:09:30] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Create maintain-views user for labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T148560#2757157 (10chasemp) thank you @jcrespo! fyi this is maintained here atm (both user and pass are set in private) https://phabricator.wikimedia.org/diffus... [16:14:37] 06Labs, 15User-Hydriz: Dumps instances occasionally hammer NFS for temporary storage - https://phabricator.wikimedia.org/T134148#2757173 (10chasemp) We do have the ability I believe to throttle this client side natively using tc. We do this in all cases currently. I'm not actually sure why this is still hamm... [16:19:51] 06Labs: Puppet tab in Horizon unusably slow - https://phabricator.wikimedia.org/T149589#2757207 (10yuvipanda) [16:20:35] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Create maintain-views user for labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T148560#2757223 (10jcrespo) I feel there is another missunderstanding, there is $::passwords::mysql::maintain_views and $::passwords::labsdb::maintainviews. I wil... [16:22:09] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: Migrate labsdb1005/1006/1007 to jessie - https://phabricator.wikimedia.org/T123731#2757233 (10chasemp) >>! In T123731#2757001, @yuvipanda wrote: > If we settle on a date and announce on labs-announce... @yuvipanda I think the asks h... [16:22:41] 06Labs: Puppet tab in Horizon unusably slow - https://phabricator.wikimedia.org/T149589#2757207 (10Volans) I can confirm all the slowness, in particular in the Puppet-related stuff on the Horizon UI but also in general in the Horizon UI, from the login to each page change. [16:23:35] 06Labs, 10Labs-Infrastructure, 10DBA: Move dbproxy1010 and dbproxy1011 to labs-support network, rename them to labsdbproxy1001 and labsdbproxy1002 - https://phabricator.wikimedia.org/T149170#2757249 (10chasemp) @jynus great thanks :) fyi reminder on a pro/con task for discussion re: proxysql vs haproxy :) [16:26:10] 06Labs, 10Labs-Infrastructure, 06Services (watching): Novaproxy in labs strips out the ETag header if gzip is enabled - https://phabricator.wikimedia.org/T148676#2757258 (10yuvipanda) So, this is about gzip at the nginx level for proxied content. Looking at http://nginx.org/en/docs/http/ngx_http_gzip_module.... [16:32:22] PROBLEM - Puppet run on tools-exec-1403 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [16:33:23] PROBLEM - Puppet run on tools-exec-1202 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:33:27] PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:33:35] PROBLEM - Puppet run on tools-bastion-05 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:43:27] RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:48:30] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Create maintain-views user for labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T148560#2757344 (10chasemp) ok thanks, `$::passwords::labsdb::maintainviews` works for me [16:53:45] 06Labs, 10Labs-Infrastructure, 06Services (watching): Novaproxy in labs strips out the ETag header if gzip is enabled - https://phabricator.wikimedia.org/T148676#2757359 (10Pchelolo) The working URI to test is: `https://appservice.wmflabs.org/en.wikipedia.beta.wmflabs.org/v1/page/mobile-sections/User:Pchelolo` [16:57:22] !log deployment-prep Added Niharika29 as project member [16:57:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [16:57:33] Niharika: ^ [16:58:25] I don't think you will need admin (sudo) rights to run mwscript things, but if it turns out that you do let me know and I'll give you more rights [16:59:21] 06Labs, 10Labs-Infrastructure, 06Services (watching): Novaproxy in labs strips out the ETag header if gzip is enabled - https://phabricator.wikimedia.org/T148676#2757381 (10yuvipanda) Hitting upstream directly, etag works: ``` curl -H 'accept-encoding: gzip' -I -H 'host: appservice.wmflabs.org' http://appse... [17:03:48] !log tools.lolrrit-wm testing some custom commands for grrrit-wm, starting with !grrrit-wm-die command as a starter. [17:03:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lolrrit-wm/SAL [17:04:23] die command i love it paladox [17:04:39] It is a test command and will have a better name later [17:04:50] no i love like the name xD [17:06:25] no comment still about the test instance? [17:12:23] RECOVERY - Puppet run on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [17:13:25] RECOVERY - Puppet run on tools-exec-1202 is OK: OK: Less than 1.00% above the threshold [0.0] [17:13:35] RECOVERY - Puppet run on tools-bastion-05 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:28] yuvipanda can i have temporary access to help paladox out so s/he isnt doing all the lolrrit changes on their own? [17:25:50] Zppix: I'm no longer maintainer of grrrit-wm, do ask people who are :) [17:26:42] yuvipanda do i have you permission to add him please? [17:27:37] awww :D [17:27:47] feel free to, but really, I'm just a normal bystander now :) [17:28:04] yuvipanda no, your a panda get it right xD [17:28:05] Ok thanks :) [17:28:21] Zppix : it could be a bystanding panda [17:28:39] which, to be fair, is pretty much what all pandas are anyway [17:28:50] Alphos no pandas deserve everything xD there are so adorable [17:29:20] they're still bears, they can tear you limb from limb [17:29:30] red pandas, on the other hand, is where cuteness is at [17:29:46] Alphos yuvipanda :/ i call that panda racism xD [17:29:57] it's specism [17:30:48] red pandas are scientifically cuter than giant pandas, period https://i.ytimg.com/vi/b6dT4kyVUuY/maxresdefault.jpg [17:31:17] yuvipanda cover your ears i still think your species is better xD [17:32:01] !log tools.lolrrit-wm adding Zppix as project user [17:32:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lolrrit-wm/SAL [17:43:53] 06Labs, 10Tool-Labs: Tools puppet runs hanging - https://phabricator.wikimedia.org/T148244#2757613 (10Andrew) 05Open>03Resolved a:03Andrew [17:49:11] 06Labs, 10Labs-Infrastructure, 06Services (watching): Novaproxy in labs strips out the ETag header if gzip is enabled - https://phabricator.wikimedia.org/T148676#2729694 (10GWicke) Varnish keeps the ETag header intact, independent of gzip or not. While this might not work perfectly with byte range requests,... [17:51:50] 06Labs, 06Operations, 06Research-and-Data-Backlog, 10hardware-requests: eqiad: 2 hardware access request for research labsdbs - https://phabricator.wikimedia.org/T146065#2757635 (10RobH) [17:52:05] 06Labs, 06Operations, 06Research-and-Data-Backlog, 10hardware-requests: eqiad: 2 hardware access request for research labsdbs - https://phabricator.wikimedia.org/T146065#2649413 (10RobH) [18:02:35] 06Labs, 10Labs-Infrastructure, 06Services (watching): Novaproxy in labs strips out the ETag header if gzip is enabled - https://phabricator.wikimedia.org/T148676#2757673 (10Pchelolo) I've just tested and nginx preserves the weak etags, and according to the weak etag definition, it means that the content in s... [18:03:29] 06Labs, 06Operations, 06Research-and-Data-Backlog, 10hardware-requests: eqiad: 2 hardware access request for research labsdbs - https://phabricator.wikimedia.org/T146065#2757698 (10yuvipanda) [18:04:22] 06Labs, 06Operations, 06Research-and-Data-Backlog, 10hardware-requests: eqiad: 2 hardware access request for research labsdbs - https://phabricator.wikimedia.org/T146065#2649413 (10yuvipanda) [18:05:08] 06Labs, 06Operations, 06Research-and-Data-Backlog, 10hardware-requests: eqiad: 2 hardware access request for research labsdbs - https://phabricator.wikimedia.org/T146065#2649413 (10RobH) Please note we need to have some additional rationale on why these systems will be needed, since they are high cost syst... [18:06:04] 06Labs, 06Operations, 06Research-and-Data-Backlog, 10hardware-requests: eqiad: 2 hardware access request for research labsdbs - https://phabricator.wikimedia.org/T146065#2757722 (10yuvipanda) I edited the task to have some more info on rationale. [18:14:04] 06Labs, 06Operations, 06Research-and-Data-Backlog, 10hardware-requests: eqiad: 2 hardware access request for research labsdbs - https://phabricator.wikimedia.org/T146065#2757739 (10yuvipanda) [18:16:32] 06Labs, 06Operations, 06Research-and-Data-Backlog, 10hardware-requests: eqiad: 2 hardware access request for research labsdbs - https://phabricator.wikimedia.org/T146065#2757742 (10yuvipanda) [18:16:47] 06Labs, 06Operations, 06Research-and-Data-Backlog, 10hardware-requests: eqiad: 2 hardware access request for research labsdbs - https://phabricator.wikimedia.org/T146065#2649413 (10yuvipanda) [18:22:58] 06Labs, 06Operations, 06Research-and-Data-Backlog, 10hardware-requests: eqiad: 2 hardware access request for research labsdbs - https://phabricator.wikimedia.org/T146065#2649413 (10jcrespo) > match existing labsdbs ordered on T131363 For that, we bought HDs, not full servers, but the plan was to buy HDs a... [18:28:05] 06Labs, 10Labs-Infrastructure, 06Services (watching): Novaproxy in labs strips out the ETag header if gzip is enabled - https://phabricator.wikimedia.org/T148676#2757804 (10GWicke) Neither weak nor strong validators capture "byte identical before optional compression" well. https://tools.ietf.org/html/rfc723... [18:38:22] don't type cat lolrrrit logs in shell... [19:01:07] 06Labs, 06Operations, 06Research-and-Data-Backlog, 10hardware-requests: eqiad: 2 hardware access request for research labsdbs - https://phabricator.wikimedia.org/T146065#2757960 (10yuvipanda) @jcrespo my understanding of what was communicated to you (both at the offsite and other non-phabricator venues) wa... [19:05:25] PROBLEM - Host tools-worker-1005 is DOWN: CRITICAL - Host Unreachable (10.68.23.47) [19:21:42] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Create maintain-views user for labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T148560#2758017 (10jcrespo) a:05jcrespo>03None So this are the privileges created on all labsdbs (not yet on 9/10/11), but on 8 and the existing labs dbs: {P... [19:21:45] (03Draft1) 10Paladox: Adds a grrrit-wm restarting command for you to type in irc [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) [19:21:48] (03Draft2) 10Paladox: Adds a grrrit-wm restarting command for you to type in irc [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) [19:22:47] (03CR) 10Zppix: [C: 031] Adds a grrrit-wm restarting command for you to type in irc [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox) [19:30:43] !grrit-wm-hi [19:30:46] !grrrit-wm-hi [19:33:10] 06Labs, 06Operations, 06Research-and-Data-Backlog, 10hardware-requests: eqiad: 2 hardware access request for research labsdbs - https://phabricator.wikimedia.org/T146065#2758047 (10jcrespo) Arbitrary access clarified, I still see as new serving extra datasets that was not part of the original communication... [19:37:19] PROBLEM - Puppet run on tools-docker-builder-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:43:59] 06Labs, 10Beta-Cluster-Infrastructure: Move deployment-prep to role::puppetmaster::standalone - https://phabricator.wikimedia.org/T149620#2758099 (10yuvipanda) [19:44:23] 06Labs, 10Beta-Cluster-Infrastructure: Move deployment-prep to role::puppetmaster::standalone - https://phabricator.wikimedia.org/T149620#2758111 (10yuvipanda) Instructions in https://wikitech.wikimedia.org/wiki/Standalone_puppetmaster#Step_2:_Setup_a_puppet_client [19:44:50] 06Labs, 10Beta-Cluster-Infrastructure: Move deployment-prep to role::puppetmaster::standalone - https://phabricator.wikimedia.org/T149620#2758112 (10AlexMonk-WMF) a:03AlexMonk-WMF [19:45:27] 06Labs, 10Beta-Cluster-Infrastructure: Move deployment-prep to role::puppetmaster::standalone - https://phabricator.wikimedia.org/T149620#2758113 (10yuvipanda) When building new puppetmaster, we don't want it to have to use the current delpoyment-prep puppetmaster. So we should set a hiera variable: ``` puppe... [20:15:52] 06Labs, 10Beta-Cluster-Infrastructure: Move deployment-prep to role::puppetmaster::standalone - https://phabricator.wikimedia.org/T149620#2758219 (10AlexMonk-WMF) [20:24:17] 06Labs, 10Labs-Infrastructure, 06Operations, 07Wikimedia-Incident: Some labs instances IP have multiple PTR entries in DNS - https://phabricator.wikimedia.org/T115194#2758249 (10yuvipanda) This continues to cause issues. Clush doesn't work from tools-puppetmaster-02, at least partially because: ``` Oct 31... [20:57:21] !log deployment-prep moving some nodes to deployment-puppetmaster02 [20:57:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [21:02:12] chasemp: can we add a check here that bails if 'primary' exists? this means checking both nodes right? should we just check for presence of string Primary anywhere in drbdadm role all [21:02:38] I'm missing context I think madhuvishy [21:02:56] but no I don't agree I think in that we don't just want to know any old node is primary [21:02:59] chasemp: ah this was a comment you had pointed out in my patch - https://gerrit.wikimedia.org/r/#/c/318963/3/modules/role/templates/labs/nfs/nfs-manage.sh.erb [21:03:10] sorry i wasn't clear [21:03:40] check may be a loaded word [21:04:06] basically if you try to issue start/up on a node and something is already primary it could say no thanks [21:04:07] cheque is definately loaded [21:04:49] madhuvishy: ignorable for now if you are in the middle of things, it's an easily reasoned about and addable nicety [21:05:47] chasemp: right, i was asking - if we look through the output of drbdadm role all and see the word Primary - then we should bail? [21:06:24] that seems like a big hammer way to do it, if all things travel together it seems reasonable [21:06:55] yeah, we don't really separate this script per resource though [21:07:37] that's ok though [21:07:50] as we don't want to separate per now anyhow [21:08:07] if we did then it would mean it's not true we could flag errant on 'Primary' being found [21:08:16] right [21:08:19] but now we can [21:08:22] agreed [21:08:47] I usually bullish in prevent-humans-from-being-humans type checks like this but in thsi case [21:08:50] maybe adviseable [21:15:22] yuvipanda: thank you for inventing kubernetes [21:17:08] screenshotted and hung on my wall [21:20:18] I like calling people the inventor of things. I once introduced a guy at a meeting as "the inventor of the railroad" because he had an interest in railroads. [21:21:45] I look forward to the day it's done in front the actual inventor inadvertently and the awkwardness that ensues [21:22:36] granted railroad is very funny [21:31:06] hare: haha [21:31:11] hare: thank you for inventing wikiprojects :D [21:31:16] :O [21:31:35] yuvipanda: kubernetes allows me to run a flask app on tool labs, something i didn't think was possible [21:31:40] hare: :D [21:31:54] it was already, but was just really slow and cumbersome [21:44:13] !log tools restarted cron on tools-cron-01 [21:44:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:50:11] !log tools deleted cyberbot queue with qconf -dq cyberbot [21:50:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:56:47] 06Labs, 10Tool-Labs: Data collection for tools job_count seems to be broken - https://phabricator.wikimedia.org/T149634#2758523 (10bd808) [21:57:32] 06Labs, 10Tool-Labs: Data collection for tools job_count seems to be broken - https://phabricator.wikimedia.org/T149634#2758537 (10bd808) [22:05:51] 06Labs, 10Tool-Labs: Data collection for tools job_count seems to be broken - https://phabricator.wikimedia.org/T149634#2758523 (10chasemp) let's see if this works > root@tools-bastion-03:~# qconf -de tools-exec-cyberbot.eqiad.wmflabs > root@tools-bastion-03.tools.eqiad.wmflabs removed "tools-exec-cyberbot.eq... [22:36:48] 10Tool-Labs-tools-Other: create tool to crunch metrics for views (play started) of video and audio files - https://phabricator.wikimedia.org/T116363#2758657 (10harej-NIOSH) The API documentation is located at P4339 if you are interested in developing tools around these metrics. (Pinging @MusikAnimal.) Note that... [22:43:27] 10Tool-Labs-tools-Other: create tool to crunch metrics for views (play started) of video and audio files - https://phabricator.wikimedia.org/T116363#2758663 (10MusikAnimal) >>! In T116363#2758657, @harej-NIOSH wrote: > The API documentation is located at P4339 if you are interested in developing tools around the... [22:45:18] 10Tool-Labs-tools-Other: create tool to crunch metrics for views (play started) of video and audio files - https://phabricator.wikimedia.org/T116363#2758667 (10harej-NIOSH) >>! In T116363#2758663, @MusikAnimal wrote: >So this goes off of the raw dumps at https://dumps.wikimedia.org/other/mediacounts/ ? Correct.... [22:48:43] (03CR) 10Paladox: [C: 04-1] "This is only a test and will improve it :)" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox) [22:49:10] (03CR) 10Paladox: "test" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox) [23:02:15] (03PS3) 10Paladox: Adds a grrrit-wm restarting command for you to type in irc [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) [23:11:12] (03CR) 10Reedy: [C: 04-1] Adds a grrrit-wm restarting command for you to type in irc (031 comment) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox) [23:12:08] 10Tool-Labs-tools-Pageviews: Add Medaiviews to Pageviews suite - https://phabricator.wikimedia.org/T149642#2758721 (10MusikAnimal) [23:13:16] . [23:13:32] 10Tool-Labs-tools-Pageviews: Add Medaiviews to Pageviews suite - https://phabricator.wikimedia.org/T149642#2758737 (10MusikAnimal) [23:14:33] (03CR) 10Peachey88: [C: 04-1] "I don't see the need for this at all, Grrrit-wm shouldn't be getting restarted enough where a irc based command is required." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox) [23:16:15] (03CR) 10Zppix: "We're still working on it also, this is just part 1 of a even bigger feature that we have in mind" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox) [23:17:02] (03CR) 10Zppix: Adds a grrrit-wm restarting command for you to type in irc (031 comment) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox) [23:17:26] (03CR) 10BryanDavis: "> Patch Set 2: Code-Review+1" (032 comments) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox) [23:19:07] (03CR) 10Reedy: Adds a grrrit-wm restarting command for you to type in irc (031 comment) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox) [23:25:50] (03CR) 10Legoktm: "This seems misguided. We need to restart the gerrit stream events listener when Gerrit is restarted, not the IRC bot component." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox) [23:35:23] (03CR) 10Zppix: Adds a grrrit-wm restarting command for you to type in irc (033 comments) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox) [23:36:10] (03CR) 10Zppix: "> This seems misguided. We need to restart the gerrit stream events" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox) [23:53:08] (03CR) 10Paladox: "@Peachey88 @Reedy @Legoktm, @BryanDavis all of this is working progress, I'm testing so the messages will look off to a production irc net" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox) [23:54:17] (03CR) 10Reedy: "You can't || a load of strings like that. It just won't work" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox) [23:55:07] (03CR) 10Paladox: "> You can't || a load of strings like that. It just won't work" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox)