Loading...

nunit-discuss@googlegroups.com

[Prev] Thread [Next]  |  [Prev] Date [Next]

[nunit-discuss] Re: Only can run entire suite of tests on 2.6 nickdu Wed Apr 04 09:05:29 2012

I'm going to put some effort in today to get a simple repro of this.
I hope I'm successful.  By the way, I tried 2.5.10 and the same issue
occurs so it's not specific to 2.6.

Thanks,
Nick

On Apr 2, 5:09 pm, Charlie Poole <[EMAIL PROTECTED]> wrote:
> That would help. I'd definitely like to see it.
>
> Regarding dependencies, I'd say that if running tests 1-4 has an
> effect on test 5, that's a dependency by definition. It's just one that
> we don't yet understand and which is probably due to the underlying
> platform - or NUnit itself - rather than something you coded explicitly.
>
> Charlie
>
>
>
> On Mon, Apr 2, 2012 at 12:43 PM, nickdu <[EMAIL PROTECTED]> wrote:
> > Yeah, I'm thinking about spending a little bit of time to create a
> > skelaton of our framework that has the same dependent assemblies to
> > see if I can repro and if so send it along or post the code.
>
> > With respect to test dependencies, I don't think this could be the
> > case.  As I said, I can start NUnit GUI fresh and run Lock Test5
> > without a problem, and therefore it's not dependent on running any
> > other tests.  However, if I start NUnit GUI fresh and select the Lock
> > 'set' of tests and run them then Test1 - Test4 succeed and Test5 -
> > Test8 succeed.
>
> > Nick
>
> > On Apr 2, 12:54 pm, Charlie Poole <[EMAIL PROTECTED]> wrote:
> >> Hi Nick,
>
> >> On Mon, Apr 2, 2012 at 6:04 AM, nickdu <[EMAIL PROTECTED]> wrote:
> >> > There were trace files created with trace information in them, just
> >> > none of it seemed to point to a reason for the issue I'm having.
>
> >> Right - that's what I meant. NUnit is simply considering your
> >> problem as a test failure, not an NUnit problem worthy of logging.
>
> >> > I don't have any automatic reload setting turned on.
>
> >> OK
>
> >> > When I mentioned loading it was more about how NUnit loads a class and/
> >> > or method in order to execute it.  It seems odd that if I select the
> >> > Lock class and run I get different behavior than when I choose each
> >> > Lock test method individually and run them.  My guess is that NUnit is
> >> > loading the code differently in those two cases.
>
> >> NUnit's "Loading" involves only your test assembly. It has to happen before
> >> the tests can be displayed in the Gui. That loading is only repeated if you
> >> use a reload item on the menu or if you have enabled automatic reload on
> >> run or when the assembly changes.
>
> >> Loading of dependent assemblies happens according to normal .NET rules,
> >> as the code is jitted and executed. Again, once a dependent assembly is
> >> loaded into the test AppDomain, it stays there.
>
> >> So, your problem arises when the dependent assembly is first needed by
> >> one of the Lock tests. If you have run some other test first, which 
> >> successfully
> >> loads the assembly, then the Lock tests have no problem. This is exactly
> >> the sort of inter-test dependency I was originally writing about - not
> >> necessarily
> >> a dependency that you intentionally created, but one that exists 
> >> nevertheless.
>
> >> I guess that gives you two possible attacks on the problem:
> >> 1. Trace how and why the Lock class cannot load the assembly.
> >> 2. Trace how and why the complete class is able to load the assembly.
>
> >> What's different in these classes?
>
> >> Charlie
>
> >> PS: Feel free to post some code if it doesn't expose any company secrets.
>
> >> > Thanks,
> >> > Nick
>
> >> > On Apr 1, 6:08 pm, Charlie Poole <[EMAIL PROTECTED]> wrote:
> >> >> Well, the absence of trace info from NUnit at least tells us that NUnit 
> >> >> is
> >> >> not aware of any problem. I guess we knew that though.
>
> >> >> Regarding loading: NUnit does not reload any tests unless you tell it 
> >> >> to.
> >> >> You should probably turn off any automatic reload settings if you have
> >> >> them set. That way you can manually control when a reload takes place.
>
> >> >> Charlie
>
> >> >> On Sun, Apr 1, 2012 at 2:55 PM, nickdu <[EMAIL PROTECTED]> wrote:
> >> >> > I haven't use NUnit's tracing yet.  I just now tried turning on the
> >> >> > most verbose level of tracing.  I looked through all the trace files
> >> >> > generated and there didn't seem to be any information in there which
> >> >> > pointed to the problem.
>
> >> >> > Still doing more testing.  As I mentioned before, if I select the Lock
> >> >> > set of tests, and Lock is a class with a bunch of test methods, and
> >> >> > run that set, the first four tests, test1 - test4, succeed and the
> >> >> > last four, test5 - test8, fail.  The first four are negative test
> >> >> > cases which don't make it past the first assembly implementing our API
> >> >> > as it validates the parameters and fails.  The last four have valid
> >> >> > parameters and thus make it farther into our API and require this
> >> >> > Common assembly be loaded.  This is where it fails.
>
> >> >> > Interestingly, I can execute test1 successfully and then select test5
> >> >> > and execute that successfully in the same instance of the NUnit GUI.
> >> >> > However if I select all Lock test cases it fails.  Does NUnit load
> >> >> > things differently if you select individual tests version a set of
> >> >> > tests by choosing the class?
>
> >> >> > Thanks,
> >> >> > Nick
>
> >> >> > On Apr 1, 1:24 pm, Charlie Poole <[EMAIL PROTECTED]> wrote:
> >> >> >> Still, this doesn't explain why your tests have anything at all to do
> >> >> >> with the primary AppDomain. NUnit does run all tests in a separate
> >> >> >> AppDomain and creates a new one each time you reload the tests.
>
> >> >> >> Have you turned on NUnit's internal trace and looked at whether it
> >> >> >> tells you anything?
>
> >> >> >> Charlie
>
> >> >> >> On Sun, Apr 1, 2012 at 5:50 AM, nickdu <[EMAIL PROTECTED]> wrote:
> >> >> >> > I did some more tests, and while I found out some new behavior I 
> >> >> >> > still
> >> >> >> > don't know what's going on here.
>
> >> >> >> > In order to test your custom exception suggestion I figured I would
> >> >> >> > take one of the tests that failed and catch all exceptions and turn
> >> >> >> > them into an InvalidOperationException, e.g like the following:
>
> >> >> >> > catch(Exception e)
> >> >> >> >   {
> >> >> >> >   throw(new InvalidOperationException(string.Format("Caught an
> >> >> >> > exception: {0}", e.ToString())));
> >> >> >> >   }
>
> >> >> >> > I then ran just that test and much to my surprise it succeeded.  So
> >> >> >> > then I commented out the try/catch I added and rebuilt and 
> >> >> >> > retested,
> >> >> >> > just that test, and again it succeeded.  I tested, individually 
> >> >> >> > which
> >> >> >> > a fresh new instance of NUnit GUI, each test that failed and they 
> >> >> >> > all
> >> >> >> > succeeded.  For lock I have tests1 - test8.  When I run all lock
> >> >> >> > tests, test1 - test4 succeed and test5 - test8 fail.  If I run 
> >> >> >> > test5 -
> >> >> >> > test8 individually with a fresh instance of NUnit GUI they all
> >> >> >> > succeed.  The reason for running fresh instances of NUnit GUI is
> >> >> >> > because it seems once Common gets loaded then all tests from them 
> >> >> >> > on
> >> >> >> > will work as expected.  Also, once loading Common fails it will
> >> >> >> > continue to fail for that instance of NUnit GUI (looks like a type
> >> >> >> > initializer, e.g. static constructor, fails and once that happens 
> >> >> >> > the
> >> >> >> > there's no fixing it until the appdomain is unloaded).
>
> >> >> >> > Thanks,
> >> >> >> > Nick
>
> >> >> >> > On Apr 1, 8:16 am, nickdu <[EMAIL PROTECTED]> wrote:
> >> >> >> >> While we do throw some custom exceptions, my initial guess is that
> >> >> >> >> this is not what's going on here because some of the tests that 
> >> >> >> >> are
> >> >> >> >> failing are positive tests, ones which shouldn't be throwing any
> >> >> >> >> exceptions.
>
> >> >> >> >> I just ran another scenario to maybe shed some more light on the
> >> >> >> >> issue, though it doesn't shed much more light.  I have a class 
> >> >> >> >> called
> >> >> >> >> Complete which has a bunch of tests which test our Complete 
> >> >> >> >> method.
> >> >> >> >> These tests always work.  I have another class called Lock which 
> >> >> >> >> tests
> >> >> >> >> our Lock method.  Lock doesn't work if run by itself (e.g. only
> >> >> >> >> itself) after just starting the NUnit GUI.  I mentioned originally
> >> >> >> >> that if I run all the tests these other ones that fail by 
> >> >> >> >> themselves
> >> >> >> >> will succeed.  That's not entirely true.  If I start NUnit GUI up
> >> >> >> >> fresh and then run the Complete tests I can then run the Lock 
> >> >> >> >> tests,
> >> >> >> >> using the currently running NUnit GUI that ran the Complete tests,
> >> >> >> >> successfully.
>
> >> >> >> >> Here is what my guess was as to what's going on.  It appears the
> >> >> >> >> tests, and I mean the code in the NUnit test methods, which have 
> >> >> >> >> code
> >> >> >> >> which directly references code from the Common assembly, this is 
> >> >> >> >> the
> >> >> >> >> assembly that's having the issue of getting loaded, will always 
> >> >> >> >> work
> >> >> >> >> whether run separately or when running all tests.  The tests which
> >> >> >> >> don't reference code from the Common assembly won't work unless 
> >> >> >> >> the
> >> >> >> >> Common assembly has already been loaded into the process by first
> >> >> >> >> running a test that referenced it.  Almost as if the code is 
> >> >> >> >> getting
> >> >> >> >> jitted with an app base setting which is correct and allows the 
> >> >> >> >> .NET
> >> >> >> >> assembly loader to find the Common assembly using the usual 
> >> >> >> >> probing
> >> >> >> >> mechanism, but then executed in a different appdomain which 
> >> >> >> >> doesn't
> >> >> >> >> have the correct app base setting such that if/when Common needs 
> >> >> >> >> to be
> >> >> >> >> loaded it cannot be found.
>
> >> >> >> >> Thanks,
> >> >> >> >> Nick
>
> >> >> >> >> On Mar 31, 3:15 pm, Charlie Poole <[EMAIL PROTECTED]> wrote:
>
> >> >> >> >> > Hi Nick,
>
> >> >> >> >> > From the error below, it's clear that NUnit itself - rather 
> >> >> >> >> > than your
> >> >> >> >> > tests - is trying to load the dll in question. This usually 
> >> >> >> >> > happens
> >> >> >> >> > when a custom exception, defined in one of your assemblies, is
> >> >> >> >> > uncaught in the tests. Is that a possibility in this scenario?
>
> >> >> >> >> > If this is the case, then NUnit would  be failing in 
> >> >> >> >> > deserializing the
> >> >> >> >> > exceptioni because it can't load the assembly in which the 
> >> >> >> >> > exception
> >> >> >> >> > is defined. One way to find out is to temporarily copy the 
> >> >> >> >> > "missing"
> >> >> >> >> > assembly to the directory containing nunit-agent.exe, along 
> >> >> >> >> > with any
> >> >> >> >> > dependencies that may be needed.
>
> ...
>
> read more »

-- 
You received this message because you are subscribed to the Google Groups 
"NUnit-Discuss" group.
To post to this group, send email to [EMAIL PROTECTED]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/nunit-discuss?hl=en.