Loading...

nunit-discuss@googlegroups.com

[Prev] Thread [Next]  |  [Prev] Date [Next]

Re: [nunit-discuss] Re: Only can run entire suite of tests on 2.6 Charlie Poole Mon Apr 02 15:01:13 2012

That would help. I'd definitely like to see it.

Regarding dependencies, I'd say that if running tests 1-4 has an
effect on test 5, that's a dependency by definition. It's just one that
we don't yet understand and which is probably due to the underlying
platform - or NUnit itself - rather than something you coded explicitly.

Charlie

On Mon, Apr 2, 2012 at 12:43 PM, nickdu <[EMAIL PROTECTED]> wrote:
> Yeah, I'm thinking about spending a little bit of time to create a
> skelaton of our framework that has the same dependent assemblies to
> see if I can repro and if so send it along or post the code.
>
> With respect to test dependencies, I don't think this could be the
> case.  As I said, I can start NUnit GUI fresh and run Lock Test5
> without a problem, and therefore it's not dependent on running any
> other tests.  However, if I start NUnit GUI fresh and select the Lock
> 'set' of tests and run them then Test1 - Test4 succeed and Test5 -
> Test8 succeed.
>
> Nick
>
> On Apr 2, 12:54 pm, Charlie Poole <[EMAIL PROTECTED]> wrote:
>> Hi Nick,
>>
>> On Mon, Apr 2, 2012 at 6:04 AM, nickdu <[EMAIL PROTECTED]> wrote:
>> > There were trace files created with trace information in them, just
>> > none of it seemed to point to a reason for the issue I'm having.
>>
>> Right - that's what I meant. NUnit is simply considering your
>> problem as a test failure, not an NUnit problem worthy of logging.
>>
>> > I don't have any automatic reload setting turned on.
>>
>> OK
>>
>> > When I mentioned loading it was more about how NUnit loads a class and/
>> > or method in order to execute it.  It seems odd that if I select the
>> > Lock class and run I get different behavior than when I choose each
>> > Lock test method individually and run them.  My guess is that NUnit is
>> > loading the code differently in those two cases.
>>
>> NUnit's "Loading" involves only your test assembly. It has to happen before
>> the tests can be displayed in the Gui. That loading is only repeated if you
>> use a reload item on the menu or if you have enabled automatic reload on
>> run or when the assembly changes.
>>
>> Loading of dependent assemblies happens according to normal .NET rules,
>> as the code is jitted and executed. Again, once a dependent assembly is
>> loaded into the test AppDomain, it stays there.
>>
>> So, your problem arises when the dependent assembly is first needed by
>> one of the Lock tests. If you have run some other test first, which 
>> successfully
>> loads the assembly, then the Lock tests have no problem. This is exactly
>> the sort of inter-test dependency I was originally writing about - not
>> necessarily
>> a dependency that you intentionally created, but one that exists 
>> nevertheless.
>>
>> I guess that gives you two possible attacks on the problem:
>> 1. Trace how and why the Lock class cannot load the assembly.
>> 2. Trace how and why the complete class is able to load the assembly.
>>
>> What's different in these classes?
>>
>> Charlie
>>
>> PS: Feel free to post some code if it doesn't expose any company secrets.
>>
>>
>>
>> > Thanks,
>> > Nick
>>
>> > On Apr 1, 6:08 pm, Charlie Poole <[EMAIL PROTECTED]> wrote:
>> >> Well, the absence of trace info from NUnit at least tells us that NUnit is
>> >> not aware of any problem. I guess we knew that though.
>>
>> >> Regarding loading: NUnit does not reload any tests unless you tell it to.
>> >> You should probably turn off any automatic reload settings if you have
>> >> them set. That way you can manually control when a reload takes place.
>>
>> >> Charlie
>>
>> >> On Sun, Apr 1, 2012 at 2:55 PM, nickdu <[EMAIL PROTECTED]> wrote:
>> >> > I haven't use NUnit's tracing yet.  I just now tried turning on the
>> >> > most verbose level of tracing.  I looked through all the trace files
>> >> > generated and there didn't seem to be any information in there which
>> >> > pointed to the problem.
>>
>> >> > Still doing more testing.  As I mentioned before, if I select the Lock
>> >> > set of tests, and Lock is a class with a bunch of test methods, and
>> >> > run that set, the first four tests, test1 - test4, succeed and the
>> >> > last four, test5 - test8, fail.  The first four are negative test
>> >> > cases which don't make it past the first assembly implementing our API
>> >> > as it validates the parameters and fails.  The last four have valid
>> >> > parameters and thus make it farther into our API and require this
>> >> > Common assembly be loaded.  This is where it fails.
>>
>> >> > Interestingly, I can execute test1 successfully and then select test5
>> >> > and execute that successfully in the same instance of the NUnit GUI.
>> >> > However if I select all Lock test cases it fails.  Does NUnit load
>> >> > things differently if you select individual tests version a set of
>> >> > tests by choosing the class?
>>
>> >> > Thanks,
>> >> > Nick
>>
>> >> > On Apr 1, 1:24 pm, Charlie Poole <[EMAIL PROTECTED]> wrote:
>> >> >> Still, this doesn't explain why your tests have anything at all to do
>> >> >> with the primary AppDomain. NUnit does run all tests in a separate
>> >> >> AppDomain and creates a new one each time you reload the tests.
>>
>> >> >> Have you turned on NUnit's internal trace and looked at whether it
>> >> >> tells you anything?
>>
>> >> >> Charlie
>>
>> >> >> On Sun, Apr 1, 2012 at 5:50 AM, nickdu <[EMAIL PROTECTED]> wrote:
>> >> >> > I did some more tests, and while I found out some new behavior I 
>> >> >> > still
>> >> >> > don't know what's going on here.
>>
>> >> >> > In order to test your custom exception suggestion I figured I would
>> >> >> > take one of the tests that failed and catch all exceptions and turn
>> >> >> > them into an InvalidOperationException, e.g like the following:
>>
>> >> >> > catch(Exception e)
>> >> >> >   {
>> >> >> >   throw(new InvalidOperationException(string.Format("Caught an
>> >> >> > exception: {0}", e.ToString())));
>> >> >> >   }
>>
>> >> >> > I then ran just that test and much to my surprise it succeeded.  So
>> >> >> > then I commented out the try/catch I added and rebuilt and retested,
>> >> >> > just that test, and again it succeeded.  I tested, individually which
>> >> >> > a fresh new instance of NUnit GUI, each test that failed and they all
>> >> >> > succeeded.  For lock I have tests1 - test8.  When I run all lock
>> >> >> > tests, test1 - test4 succeed and test5 - test8 fail.  If I run test5 
>> >> >> > -
>> >> >> > test8 individually with a fresh instance of NUnit GUI they all
>> >> >> > succeed.  The reason for running fresh instances of NUnit GUI is
>> >> >> > because it seems once Common gets loaded then all tests from them on
>> >> >> > will work as expected.  Also, once loading Common fails it will
>> >> >> > continue to fail for that instance of NUnit GUI (looks like a type
>> >> >> > initializer, e.g. static constructor, fails and once that happens the
>> >> >> > there's no fixing it until the appdomain is unloaded).
>>
>> >> >> > Thanks,
>> >> >> > Nick
>>
>> >> >> > On Apr 1, 8:16 am, nickdu <[EMAIL PROTECTED]> wrote:
>> >> >> >> While we do throw some custom exceptions, my initial guess is that
>> >> >> >> this is not what's going on here because some of the tests that are
>> >> >> >> failing are positive tests, ones which shouldn't be throwing any
>> >> >> >> exceptions.
>>
>> >> >> >> I just ran another scenario to maybe shed some more light on the
>> >> >> >> issue, though it doesn't shed much more light.  I have a class 
>> >> >> >> called
>> >> >> >> Complete which has a bunch of tests which test our Complete method.
>> >> >> >> These tests always work.  I have another class called Lock which 
>> >> >> >> tests
>> >> >> >> our Lock method.  Lock doesn't work if run by itself (e.g. only
>> >> >> >> itself) after just starting the NUnit GUI.  I mentioned originally
>> >> >> >> that if I run all the tests these other ones that fail by themselves
>> >> >> >> will succeed.  That's not entirely true.  If I start NUnit GUI up
>> >> >> >> fresh and then run the Complete tests I can then run the Lock tests,
>> >> >> >> using the currently running NUnit GUI that ran the Complete tests,
>> >> >> >> successfully.
>>
>> >> >> >> Here is what my guess was as to what's going on.  It appears the
>> >> >> >> tests, and I mean the code in the NUnit test methods, which have 
>> >> >> >> code
>> >> >> >> which directly references code from the Common assembly, this is the
>> >> >> >> assembly that's having the issue of getting loaded, will always work
>> >> >> >> whether run separately or when running all tests.  The tests which
>> >> >> >> don't reference code from the Common assembly won't work unless the
>> >> >> >> Common assembly has already been loaded into the process by first
>> >> >> >> running a test that referenced it.  Almost as if the code is getting
>> >> >> >> jitted with an app base setting which is correct and allows the .NET
>> >> >> >> assembly loader to find the Common assembly using the usual probing
>> >> >> >> mechanism, but then executed in a different appdomain which doesn't
>> >> >> >> have the correct app base setting such that if/when Common needs to 
>> >> >> >> be
>> >> >> >> loaded it cannot be found.
>>
>> >> >> >> Thanks,
>> >> >> >> Nick
>>
>> >> >> >> On Mar 31, 3:15 pm, Charlie Poole <[EMAIL PROTECTED]> wrote:
>>
>> >> >> >> > Hi Nick,
>>
>> >> >> >> > From the error below, it's clear that NUnit itself - rather than 
>> >> >> >> > your
>> >> >> >> > tests - is trying to load the dll in question. This usually 
>> >> >> >> > happens
>> >> >> >> > when a custom exception, defined in one of your assemblies, is
>> >> >> >> > uncaught in the tests. Is that a possibility in this scenario?
>>
>> >> >> >> > If this is the case, then NUnit would  be failing in 
>> >> >> >> > deserializing the
>> >> >> >> > exceptioni because it can't load the assembly in which the 
>> >> >> >> > exception
>> >> >> >> > is defined. One way to find out is to temporarily copy the 
>> >> >> >> > "missing"
>> >> >> >> > assembly to the directory containing nunit-agent.exe, along with 
>> >> >> >> > any
>> >> >> >> > dependencies that may be needed.
>>
>> >> >> >> > This issue with custom exceptions may or may not be your problem, 
>> >> >> >> > but
>> >> >> >> > it's a long-standing issue and I suggest looking at it next.
>>
>> >> >> >> > BTW, in NUnit 3.0, the plan is to prevent any exceptions from
>> >> >> >> > travelling across the remoting boundary by catching them and
>> >> >> >> > transforming them into an xml error report.
>>
>> >> >> >> > Charlie
>>
>> >> >> >> > On Sat, Mar 31, 2012 at 6:44 AM, nickdu <[EMAIL PROTECTED]> wrote:
>> >> >> >> > > It appears my assumption is somewhat correct about which 
>> >> >> >> > > methods are
>> >> >> >> > > failing.  It appears the .NET runtime is having a problem 
>> >> >> >> > > loading
>> >> >> >> > > Redbone.Framework.BusinessProcess.Common.dll.  This assembly is 
>> >> >> >> > > in the
>> >> >> >> > > same directory as my NUnit test assembly and I have all the 
>> >> >> >> > > other
>> >> >> >> > > dependent assemblies there.  This directory is 
>> >> >> >> > > c:\data\development
>> >> >> >> > > \Redbone\bpf\tests\bin\debug.  However, when the fusion loader 
>> >> >> >> > > looks
>> >> >> >> > > for the Common assembly it's looking in the app base directory 
>> >> >> >> > > where
>> >> >> >> > > the NUnit GUI .exe exists. I thought NUnit somehow changed the 
>> >> >> >> > > app
>> >> >> >> > > base (maybe by creating another appdomain and loading the tests 
>> >> >> >> > > from
>> >> >> >> > > there) to the directory where the test assembly exists?  My main
>> >> >> >> > > framework API, which all the tests make use of, gets loaded and 
>> >> >> >> > > that's
>> >> >> >> > > in the same directory as the test assembly.  Here is the log 
>> >> >> >> > > from the
>> >> >> >> > > fusion log viewer:
>>
>> ...
>>
>> read more »
>
> --
> You received this message because you are subscribed to the Google Groups 
> "NUnit-Discuss" group.
> To post to this group, send email to [EMAIL PROTECTED]
> To unsubscribe from this group, send email to [EMAIL PROTECTED]
> For more options, visit this group at 
> http://groups.google.com/group/nunit-discuss?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"NUnit-Discuss" group.
To post to this group, send email to [EMAIL PROTECTED]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/nunit-discuss?hl=en.