[00:48:59] (03PS1) 10BryanDavis: www: guard against midding $xjob->JB_hard_resource_list->qstat_l_requests [labs/toollabs] - 10https://gerrit.wikimedia.org/r/322605 [00:50:46] (03CR) 10BryanDavis: "Cherry-picked to /data/project/admin/toollabs" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/322605 (owner: 10BryanDavis) [00:52:44] !log tools.admin Cherry-picked https://gerrit.wikimedia.org/r/#/c/322605 to /data/project/admin/toollabs [00:52:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.admin/SAL [01:43:22] hello [01:43:25] KeyError: 'lgtoken' [01:43:37] i get this error [01:47:15] any help [02:33:11] PROBLEM - Puppet run on tools-exec-1413 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [03:08:09] RECOVERY - Puppet run on tools-exec-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [03:40:03] . [04:06:51] 06Labs, 10wikitech.wikimedia.org, 13Patch-For-Review: Labs front-page statistics are very wrong - https://phabricator.wikimedia.org/T139773#2809983 (10Andrew) Instance counts are now accurate! [04:41:15] PROBLEM - Puppet staleness on tools-mail-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [04:59:34] PROBLEM - Puppet staleness on tools-proxy-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [07:18:28] 06Labs, 10Tool-Labs, 10DBA, 13Patch-For-Review: Provisioning MySQL replica users fails on tool labs - https://phabricator.wikimedia.org/T151014#2810051 (10Marostegui) a:03Marostegui Thanks @yuvipanda I will leave this ticket open and ping you in two weeks! Thanks again! [07:29:02] PROBLEM - Puppet run on tools-webgrid-generic-1403 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:09:02] RECOVERY - Puppet run on tools-webgrid-generic-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [08:42:59] 06Labs, 10Labs-Infrastructure, 10DBA: Initial data tests for db1095 - https://phabricator.wikimedia.org/T150960#2810119 (10Marostegui) enwikivoyage database has been imported. I have placed the following flag in my.cnf as a temporary one: `replicate-do-db=enwikivoyage` (not added to puppet yet) Replication i... [09:58:26] RECOVERY - Host tools-secgroup-test-102 is UP: PING OK - Packet loss = 0%, RTA = 0.73 ms [10:25:38] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [11:12:07] hi i'm trying to get an account in the tool labs for my IEG project on etymological trees https://meta.wikimedia.org/wiki/Grants:IEG/A_graphical_and_interactive_etymology_dictionary_based_on_Wiktionary [11:12:39] followed the procedure in https://wikitech.wikimedia.org/wiki/Help:Tool_Labs and I'm still waiting for an account [11:12:48] any way to accelerate the process? [11:12:49] thanks a lot [11:13:03] thanks for coming to the chat [11:13:58] what project name and what account name are you requesting? mention either of the two in here, and someone might look at it [11:14:07] RECOVERY - Host tools-secgroup-test-103 is UP: PING OK - Packet loss = 0%, RTA = 1.11 ms [11:14:17] (I'm using another tool on labs, and went through the process ages ago) [11:14:46] thanks gry, my account name is Epantaleo, not sure about the project name [11:15:46] 06Labs, 10Tool-Labs: C# MONO bot runs OK but not with jsub - https://phabricator.wikimedia.org/T150099#2810333 (10Kotz) No, this does not work reliably. Sometimes it asks for key verifications. Also I understand it would not work with cron (only jsub works there). I still need some help here. Does anyone run D... [11:16:04] epantaleo, is it your first time on the chat? [11:17:59] yes [11:18:05] ok [11:18:35] about aproject name, as you are asking i google and found this page [11:18:36] t9m7tkshgn [11:18:39] sorry [11:18:42] it may take a couple hours before someone replies at some points, but you can just stick around and look, and learn which timezone is most active [11:18:47] https://phabricator.wikimedia.org/T76375 [11:19:06] yes [11:19:18] this is for labs not for tools [11:19:41] if you're making a new tool you don't need to use that task [11:20:01] the difference is: a tool is something you write yourself, often using API or a copy of database [11:20:14] a lab is something you write on top of mediawiki (such as a mediawiki extension) [11:20:27] a tool is in any language [11:20:30] a lab is usually in php [11:20:39] because mediawiki is in php [11:20:43] so yes I'm making a new tool [11:20:56] and i should just wait for an authorization at this point? [11:21:05] by looking at the URL you gave, it sounds like a tool [11:21:13] yes :) [11:22:00] what web browser are you using, and are you comfortable installing new applications on your computer? there are better ways to use this chat than the webchat [11:22:20] we could try to do that while we're waiting [11:22:46] ok [11:22:58] so what whould I do? [11:23:10] right now I'm using chrome [11:23:26] *should [11:23:28] you could try the quassel client (http://www.quassel-irc.org) and see whether you like it [11:23:47] it's free/open-source, and has zero relationship with wikimedia [11:24:02] just my personal recommendation [11:26:02] on that note, i'm heading out. I'll follow-up a few hours later [11:26:14] nice meeting you here epantaleo :) [11:26:48] thanks. just installed quassel. later :) [11:27:00] :) [11:31:53] 06Labs, 10Tool-Labs: C# MONO bot runs OK but not with jsub - https://phabricator.wikimedia.org/T150099#2810351 (10Kotz) maybe @Pengo [11:59:06] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [12:01:40] RECOVERY - Host secgroup-lag-102 is UP: PING OK - Packet loss = 0%, RTA = 0.47 ms [12:08:17] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [12:21:11] 06Labs, 10Labs-Infrastructure, 10DBA: Initial data tests for db1095 - https://phabricator.wikimedia.org/T150960#2810431 (10Marostegui) `replicate-do-db` has been changed in for: `replicate-wild-do-table = enwikivoyage.%` [12:27:40] 06Labs, 10Tool-Labs: C# MONO bot runs OK but not with jsub - https://phabricator.wikimedia.org/T150099#2773990 (10scfc) @Kotz: What do you mean by "key verifications"? Please always quote error messages verbatim. [12:58:00] 06Labs, 10Tool-Labs: C# MONO bot runs OK but not with jsub - https://phabricator.wikimedia.org/T150099#2810476 (10Kotz) @scfc, this happens only intermittently so I don't have the error message verbatim. Anyway this is not the major issue for this ticket I think. I need to be able to run by bot via jsub in ord... [12:59:19] ^ c# works better in kubectl [13:04:14] 06Labs, 10Tool-Labs: Delete probably obsolete mongo.creds.json files - https://phabricator.wikimedia.org/T151206#2810491 (10scfc) [13:19:00] PROBLEM - Puppet run on tools-worker-1012 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [13:46:10] 10Tool-Labs-tools-Other: create tool to crunch metrics for views (play started) of video and audio files - https://phabricator.wikimedia.org/T116363#2810582 (10ezachte) I'm really glad this work is happening. As for which columns to include here is some feedback: The file layout is as follows: #1: Base name... [13:58:59] RECOVERY - Puppet run on tools-worker-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [14:01:17] 06Labs, 10Labs-Infrastructure, 10DBA: Initial data tests for db1095 - https://phabricator.wikimedia.org/T150960#2810597 (10Marostegui) @jcrespo looking at my notes, I have that the script to sanitize is: `redact_standard_output.sh` Before running it tomorrow I will ping you anyways so we are on the same page... [14:04:27] 06Labs, 10Labs-Infrastructure, 10DBA: Initial data tests for db1095 - https://phabricator.wikimedia.org/T150960#2810600 (10jcrespo) Yes, but there are private tables that have to be dropped (they are on the replication filters). Also, non-regular wikis have a different process. [14:06:35] 06Labs, 10Labs-Infrastructure, 10DBA: Initial data tests for db1095 - https://phabricator.wikimedia.org/T150960#2802738 (10Krenair) Non-regular wikis? What does that refer to? [14:10:11] 06Labs, 10Labs-Infrastructure, 10DBA: Initial data tests for db1095 - https://phabricator.wikimedia.org/T150960#2810611 (10jcrespo) @Marostegui, is is replicating from s3-master? We should think a strategy to test ROW-based replication triggers (from dbstore1002? from another slave?). [14:10:20] 06Labs, 10Labs-Infrastructure, 10DBA: Initial data tests for db1095 - https://phabricator.wikimedia.org/T150960#2810612 (10Marostegui) Thanks - as per the replication filters, these tables woulld need to be dropped from `enwikivoyage` then: ``` click_tracking cu_changes cu_log echo_email_batch echo_event ec... [14:11:11] 06Labs, 10Labs-Infrastructure, 10DBA: Initial data tests for db1095 - https://phabricator.wikimedia.org/T150960#2810616 (10Marostegui) >>! In T150960#2810611, @jcrespo wrote: > @Marostegui, is is replicating from s3-master? We should think a strategy to test ROW-based replication triggers (from dbstore1002?... [14:15:42] 06Labs, 10Labs-Infrastructure, 10DBA: Initial data tests for db1095 - https://phabricator.wikimedia.org/T150960#2810621 (10Marostegui) I have enabled GTID on it right now, so maybe we can move it under another slave in codfw (so we do not need to mess with eqiad) with ROW based replication. [14:17:07] 06Labs, 10Labs-Infrastructure, 10DBA: Initial data tests for db1095 - https://phabricator.wikimedia.org/T150960#2810622 (10jcrespo) >>! In T150960#2810612, @Marostegui wrote: > Thanks - as per the replication filters, these tables woulld need to be dropped from `enwikivoyage` then: That is the part I do not... [14:19:35] 06Labs, 10Labs-Infrastructure, 10DBA: Initial data tests for db1095 - https://phabricator.wikimedia.org/T150960#2810624 (10jcrespo) > I have enabled GTID on it right now, so maybe we can move it under another slave in codfw While that could be possible (I think it could too much overhead when we get more sh... [14:24:06] 06Labs, 10Labs-Infrastructure, 10DBA: Initial data tests for db1095 (temporary db1069 - sanitarium replacement) - https://phabricator.wikimedia.org/T150960#2810631 (10jcrespo) [14:24:38] 06Labs, 10Labs-Infrastructure, 10DBA: Initial data tests for db1095 (temporary db1069 - sanitarium replacement) - https://phabricator.wikimedia.org/T150960#2810632 (10Marostegui) >>! In T150960#2810622, @jcrespo wrote: >>>! In T150960#2810612, @Marostegui wrote: >> Thanks - as per the replication filters, th... [14:28:40] 06Labs: Request creation of hound labs project - https://phabricator.wikimedia.org/T148573#2810638 (10chasemp) 05Open>03stalled [14:28:55] (03CR) 10ArthurPSmith: "Glad you caught that, looks good to me." [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/322429 (owner: 10Ricordisamoa) [14:50:56] 06Labs, 10Tool-Labs: C# MONO bot runs OK but not with jsub - https://phabricator.wikimedia.org/T150099#2810692 (10scfc) `qacct -j 139995` shows that `maxvmem` registered for that job was 457.445 MBytes. While `exit_status` is 1 and IIRC it would be some larger value for out-of-memory failures, `maxvmem` is "c... [14:56:24] 06Labs, 07Tracking: Existing Labs project quota increase requests (Tracking) - https://phabricator.wikimedia.org/T140904#2810714 (10Andrew) [14:56:26] 06Labs: Revert: request increased quota (floating ip) for "cvn" labs project - https://phabricator.wikimedia.org/T150209#2810711 (10Andrew) 05Open>03Resolved a:03Andrew reverted. Thanks @Krinkle [14:57:49] 06Labs, 10Labs-Infrastructure, 10DBA: Migrate existing labs users from the old servers, if possible using roles and start maintaining users on the new database servers, too - https://phabricator.wikimedia.org/T149933#2810718 (10chasemp) I was in a position this week to work through a few issues with create-d... [15:00:09] 06Labs, 10Labs-Infrastructure, 10DBA: Migrate existing labs users from the old servers, if possible using roles and start maintaining users on the new database servers, too - https://phabricator.wikimedia.org/T149933#2810721 (10chasemp) I think what this boils down to is: if you guys #DBA's can verify the ro... [15:13:48] 06Labs, 10Labs-Infrastructure, 10DBA: Migrate existing labs users from the old servers, if possible using roles and start maintaining users on the new database servers, too - https://phabricator.wikimedia.org/T149933#2810735 (10jcrespo) Yes, only 2 comments- * Try to avoid looping over created accounts. MyS... [15:19:32] 06Labs, 10Labs-Infrastructure, 10DBA: Migrate existing labs users from the old servers, if possible using roles and start maintaining users on the new database servers, too - https://phabricator.wikimedia.org/T149933#2810744 (10Marostegui) >>! In T149933#2810735, @jcrespo wrote: > >> Make it the canonical s... [15:29:26] (03PS1) 10BBlack: add new fake entries for new unified keys [labs/private] - 10https://gerrit.wikimedia.org/r/322675 [15:41:05] (03CR) 10BBlack: [C: 032 V: 032] add new fake entries for new unified keys [labs/private] - 10https://gerrit.wikimedia.org/r/322675 (owner: 10BBlack) [15:42:21] 06Labs, 10Labs-Infrastructure, 10DBA: Initial data tests for db1095 (temporary db1069 - sanitarium replacement) - https://phabricator.wikimedia.org/T150960#2810791 (10Marostegui) I have changed db1095 to replicate from db2057 which has ROW based binlogs: ``` MariaDB PRODUCTION s3 localhost (none) > show glo... [15:44:32] 06Labs, 10Labs-Infrastructure, 10DBA: Migrate existing labs users from the old servers, if possible using roles and start maintaining users on the new database servers, too - https://phabricator.wikimedia.org/T149933#2810800 (10chasemp) That would probably be best as the first question in dealing w/ a static... [15:58:56] does anyone know how I can add a new role in Horizon ? [15:59:29] I would like to add a role that hasn't been merged in puppet.git yet . https://horizon.wikimedia.org/project/puppet/ has a list of puppet roles for the "Project" [15:59:34] you can't [15:59:35] but it is empty and I can't find how to add to it [15:59:41] have to add the class to the host via hiera [15:59:59] so something like: [16:00:00] classes: [16:00:05] - role::something::new [16:00:06] ? [16:00:06] yep [16:00:10] that's what I do [16:00:17] thanks will try [16:00:19] it's used in one or two places around deployment-prep [16:00:37] this is a known limitation [16:00:40] 06Labs, 10Tool-Labs: C# MONO bot runs OK but not with jsub - https://phabricator.wikimedia.org/T150099#2810821 (10Kotz) @scfc thanks this did the trick for BotHasava :-) [16:00:51] there was talk of improving things [16:00:56] andrew would know where we are at with that [16:01:30] if there is a workaround, that is all I need for now :} [16:21:38] Krenair: the hiera classes trick works like a charm. Thank you [16:45:16] KeyError: 'lgtoken' [16:49:13] i get this KeyError: 'lgtoken' [16:49:17] hi newbot [16:49:22] hi [16:49:23] what does this have to do with labs? [16:49:58] i don't know :D [16:50:18] when i most go for the error tools [16:53:59] Help:Tool_Labs #Troubleshooting please feel free to come into #wikimedia-labs connect [16:55:42] newbot: please frame your question something like this: I am trying to do x and to do x I have done y and y is not working for me and shows the following message [16:55:58] and be prepared to persist this to the labs-l list if none of the folks in attendance currently know [16:57:24] It sounds like newbot is trying to authenticate to the action api (api.php) and getting an error message. More details would be helpful including the language and possibly library that is being used and what exact api requests are being made. [16:57:52] lgtoken is part of the deprecated password authentication data sent to the action api [16:58:34] compat [16:59:24] newbot: so you are using pywikibot's compat branch and having authentication issues? [16:59:39] yes [17:00:32] newbot: the best place to get help on that is probaby the #pywikibot irc channel or the pywikibot mailing list [17:01:03] thanks [17:05:18] Does anyone know where on wikitech we still have a link to the "shell request" form? I see 3 new requests for that right from the last few days. [17:05:57] links to the form itself? [17:06:03] I mean, requests for links to the form itself? [17:06:16] no, filled in forms [17:06:26] e.g. https://wikitech.wikimedia.org/wiki/Shell_Request/Arian_Writing?rcid=1487255 [17:07:29] hmm, I think tim landscheidt has been doing that for a long time and I'm not exactly sure (I thought at one point it was automated but possible that was just proposed) [17:08:11] shell access is automated now, which is why I want to find out how people are still finding the form and remove that [17:08:29] you get membership in the bastion group when you are added to another project [17:08:43] chasemp: that's the tools request form that Tim L does [17:09:57] bd808: I finally understand but am not sure either [17:10:23] I'll ask the folk who filled it in on their talk pages when I tell them its not needed. :) [17:10:52] https://wikitech.wikimedia.org/wiki/Portal:Tool_Labs [17:11:01] says Fill out an access request to join the Tools project. [17:11:06] linking to https://wikitech.wikimedia.org/wiki/Special:FormEdit/Tools_Access_Request [17:11:19] yeah, that one is good [17:12:46] this one is old junk -- https://wikitech.wikimedia.org/wiki/Special:FormEdit/Shell_Access_Request [17:12:58] I suppose I can just delete the SMW form [17:40:42] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Salgo60 was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1002450 edit summary: [17:42:35] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/MnemonicFlow was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1002456 edit summary: [17:45:08] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Epantaleo was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1002463 edit summary: [17:48:49] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Isomorphyc was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1002466 edit summary: [17:50:53] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, and 3 others: Move dbproxy1010 and dbproxy1011 to labs-support network, rename them to labsdbproxy1001 and labsdbproxy1002 - https://phabricator.wikimedia.org/T149170#2811283 (10Cmjohnson) I can move these to rack c5. Can these be moved anytime or do the... [17:53:01] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/ChongDae was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1002469 edit summary: [17:54:45] (03CR) 10Ricordisamoa: "Self-merged only because the tool was broken :)" [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/322429 (owner: 10Ricordisamoa) [18:19:52] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, and 3 others: Move dbproxy1010 and dbproxy1011 to labs-support network, rename them to labsdbproxy1001 and labsdbproxy1002 - https://phabricator.wikimedia.org/T149170#2811430 (10jcrespo) They can be moved at any time, I will schedule downtime now on icinga. [19:19:37] bd808: https://wikitech.wikimedia.org/wiki/Help_talk:Tool_Labs/Right_to_fork_policy#footer [19:48:56] !log tools restart all webservice jobs on gridengine to pick up logging again [19:49:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:54:14] hi, what is the Labs IP range? I need it for OAuth registration [19:54:33] Krinkle: thanks for asking the question. I tried to respond by may have rambled. [19:54:39] *but [19:55:50] bd808: Thanks, sounds good. [20:09:37] PROBLEM - Host newtestvm2 is DOWN: CRITICAL - Host Unreachable (10.68.22.198) [20:09:41] PROBLEM - Host newtestvm is DOWN: CRITICAL - Host Unreachable (10.68.17.40) [20:22:02] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:33:37] PROBLEM - Puppet run on tools-docker-builder-03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [21:13:20] RECOVERY - Puppet run on tools-docker-builder-03 is OK: OK: Less than 1.00% above the threshold [0.0] [21:15:32] !log tools disable puppet everywhere [21:15:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:22:03] 10Tool-Labs-tools-Other: create tool to crunch metrics for views (play started) of video and audio files - https://phabricator.wikimedia.org/T116363#2812165 (10harej-NIOSH) That is correct @ezachte, or at least those are the columns I went with (so I hope they're correct!). [21:48:23] For some reason my tool's venv lacks a python3 executable?? [21:48:40] the dir in question is /data/project/mediaplaycounts/www/python/venv/bin [21:48:55] Harej: yes, I noticed that too [21:48:57] not sure why [21:49:05] I know it *used* to be there. [21:49:14] Should I just try reinstalling the whole thing? [21:49:15] Was it made from a k8s shell? [21:49:20] I believe so [21:50:01] oh wow. it doesn't even have a bin/python [21:50:14] yah probably best to start over [21:50:25] * Harej gets the flamethrower out [21:50:41] * bd808 provides supporting fire [21:54:18] And I don't want to do this within a venv, since I will be creating a venv as part of the process. Yes? [21:55:29] Harej: yeah, use the system python to create the empty venv itself [21:56:24] there are some basic instructions at https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web/Kubernetes#python_.28uwsgi_.2B_python3.4.29 [22:35:47] https://tools.wmflabs.org/phetools/statistics.php [22:35:59] what happens? [22:38:45] https://tools.wmflabs.org/bub/ [22:38:47] yannf: I just restarted it, it's up [22:39:10] I saw some SQL errors in error.log, not sure if that's related? [22:39:26] YuviPanda, ok, thanks, and for BuB? [22:39:44] doing so now [22:45:09] 06Labs, 10Tool-Labs: toollabs: puppet super broken on tool-services-01 and -02 - https://phabricator.wikimedia.org/T151264#2812591 (10Andrew) [22:46:33] 06Labs, 07LDAP: Clean up ldap host entries and references - https://phabricator.wikimedia.org/T148781#2812608 (10Andrew) [22:46:35] 06Labs, 10Tool-Labs, 13Patch-For-Review: Tools: Migrate puppet roles from ldap to the new puppetbackend - https://phabricator.wikimedia.org/T148683#2812606 (10Andrew) 05Open>03Resolved I created a bunch of tools- puppet prefixes in horizon, removed all of the per-node tools hiera files on wikitech, and p... [22:50:02] YuviPanda, could you also restart phe's bot? [22:50:28] yannf: sorry I can only restart webservices. you need someone who is a member of that tool to restart bots [22:50:35] ok [22:54:37] PROBLEM - Puppet run on tools-docker-registry-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [22:55:52] yannf: ah, bub is down and I can't do much about it because they've a custom PYTHONPATH that's preventing webservice commands from working [22:56:03] there's a bug for this, let me find [22:56:26] yannf: https://phabricator.wikimedia.org/T144554 [22:57:20] YuviPanda, ok, you got thanks from fr.wikisource ;) [23:00:16] 06Labs, 10Tool-Labs: BUB 503: AttributeError: 'module' object has no attribute 'python_2_unicode_compatible' - https://phabricator.wikimedia.org/T144554#2812649 (10yuvipanda) This just happened again. One of the maintainers needs to sort out the PYTHONPATH issue :) [23:00:27] 06Labs, 10Tool-Labs: BUB 503: AttributeError: 'module' object has no attribute 'python_2_unicode_compatible' - https://phabricator.wikimedia.org/T144554#2812650 (10yuvipanda) 05Resolved>03Open [23:01:05] yannf: I don't think there's much I can do there right now, you need to find and poke a bub maintainer [23:02:46] ok [23:03:16] BuB needs to be written again from start anyway [23:03:29] probably :) [23:03:32] yannf: sorry! [23:04:33] RECOVERY - Puppet staleness on tools-proxy-01 is OK: OK: Less than 1.00% above the threshold [3600.0] [23:14:12] YuviPanda, hi, I sent you a message [23:20:13] (I responded) [23:21:04] (03PS1) 10BryanDavis: jsub: guard against PYTHONPATH munging in caller's environment [labs/toollabs] - 10https://gerrit.wikimedia.org/r/322790 (https://phabricator.wikimedia.org/T147350) [23:21:16] RECOVERY - Puppet staleness on tools-mail-01 is OK: OK: Less than 1.00% above the threshold [3600.0] [23:22:38] (03PS2) 10BryanDavis: www: guard against missing $xjob->JB_hard_resource_list->qstat_l_requests [labs/toollabs] - 10https://gerrit.wikimedia.org/r/322605 [23:23:22] (03CR) 10jenkins-bot: [V: 04-1] jsub: guard against PYTHONPATH munging in caller's environment [labs/toollabs] - 10https://gerrit.wikimedia.org/r/322790 (https://phabricator.wikimedia.org/T147350) (owner: 10BryanDavis) [23:23:34] bd808: \o/ <3 ty [23:24:22] (03CR) 10jenkins-bot: [V: 04-1] www: guard against missing $xjob->JB_hard_resource_list->qstat_l_requests [labs/toollabs] - 10https://gerrit.wikimedia.org/r/322605 (owner: 10BryanDavis) [23:24:26] bah. f you jerkins [23:25:22] 06Labs, 10Tool-Labs: toollabs: puppet super broken on tool-services-01 and -02 - https://phabricator.wikimedia.org/T151264#2812734 (10chasemp) a:05Andrew>03chasemp I will take a look [23:27:01] (03PS2) 10BryanDavis: jsub: guard against PYTHONPATH munging in caller's environment [labs/toollabs] - 10https://gerrit.wikimedia.org/r/322790 (https://phabricator.wikimedia.org/T147350) [23:34:36] RECOVERY - Puppet run on tools-docker-registry-01 is OK: OK: Less than 1.00% above the threshold [0.0]