F# unit testing

visual studio code f# and how to use f# interactive
GregDeamons Profile Pic
GregDeamons,New Zealand,Professional
Published Date:03-08-2017
Your Website URL(Optional)
Comment
Syme_850-4C18.fm Page 523 Tuesday, October 16, 2007 12:43 PM CHA P TE R 1 8 ■ ■ ■ Debugging and Testing F Programs Anecdotal evidence indicates that functional programming frequently leads to a substantially reduced bug rate for good programmers. This is primarily because programs built using func- tional techniques tend to be highly compositional, building correct programs out of correct building blocks. Functional programming style, moreover, avoids or substantially reduces the use of side effects in the program, one property that makes programs more compositional. However, debugging and testing are still essential activities to ensure that a program is as close as possible to its specifications. Bugs and misbehaviors are facts of life, and F programmers must learn techniques to find and remove them. As a result, software testing is an important activity when developing large systems. Tests are initially carried out by simply writing small programs and interactively running them, but then a larger infrastructure quickly becomes necessary as a system grows and as new function- alities must preserve the existing ones. In this chapter, we will discuss how you can perform testing with F using F Interactive, using the debugging facilities provided by Visual Studio and the .NET infrastructure, and using the NUnit framework for unit testing. A widely adopted debugging technique is the “do-it-yourself-by-augmenting-your-program- with-printf” approach. However, this is a technique that suffers from several problems and, although still useful, should not be the only technique you are prepared to apply to the complexities associated with program testing and debugging. For testing, there are several strategies to test programs and ensure that they behave as expected, and the testing theory developed by software engineering has introduced several techniques used every day in software development. In this chapter, we focus on three aspects of program debugging and testing with F: � Using the Visual Studio debugger and the .NET debugging framework � Using F Interactive for testing and debugging � Doing unit testing using NUnit, a freely available framework for unit testing Alternative tools for debugging and unit testing are available, such as the .NET debugger that ships with the .NET Framework and the testing framework included in the Team Edition of Visual Studio. The concepts behind these tools are similar to those presented here, and the techniques discussed in this chapter can be easily adapted when using them. All these techniques 523Syme_850-4C18.fm Page 524 Tuesday, October 16, 2007 12:43 PM 524 CHAPTER 18 ■ DEBUGGING AND TESTING F PROGRAMS and tools are very helpful, but it is important to remember that these are just tools and you must use them in the appropriate way. Debugging F Programs Programming systems such as .NET support debugging as a primary activity through tools to help programmers inspect the program for possible errors. The debugger is one of the most important of these tools, and it allows you to inspect the program state during the execution. It is possible to execute the program stepwise and analyze its state during execution. DEBUGGABLE PROGRAMS The debugger requires support from debugged programs in order to work properly; for interpreted languages, it is the interpreter that supervises the program execution, and the debugger must interact with it. Compiled languages, on the other hand, must include this support during compilation so that the debugger can properly interact with the running program. The CLR provides support for program debugging, and compiled programs provide information to the debugger via a file with a .pdb file extension, which is the program debugging database. Since the compila- tion process maps high-level programming constructs into equivalent ones in a less expressive language (in this case the intermediate language), some information gets lost during this process even if the semantics of the program are preserved. An example is the name of local variables that in the intermediate language are referred to use indexes into an array rather than names. A database is used to preserve the information on the correspondence between the program instructions and the intermediate language instructions, and it is used by the debugging infrastructure to create the illusion that the program is interpreted at the language level showing the current line of execution in the source code rather than the one in the compiled and actually running program. The database retains correspondence among intermediate language instructions (and those that have been used to generate them) and other important information, such as local variable names, that is lost during compilation. The program database is language independent so that the debugger tool can be shared among different programming languages and the programmer can analyze the program execution even when a program has been developed with different languages. It is also possible to step through unmanaged code from managed code, and vice versa. Debugging without the .pdb file is still possible, though the debugger is incapable of showing the source code, and the intermediate code or the machine code is shown to the user. We’ll start with the following simple function that is, in principle, meant to return true if the input string is a palindrome and false otherwise: let isPalindrome (str:string) = let rec check(s:int, e:int) = if s = e then true elif str.s str.e then false else check(s + 1, e - 1) check(0, str.Length - 1)Syme_850-4C18.fm Page 525 Tuesday, October 16, 2007 12:43 PM CHAPTER 18 ■ DEBUGGING AND TESTING F PROGRAMS 525 The function appears correct at first sight. However, it works only for strings with an odd number of characters and strings with an even length that are not palindromes. In particular, the program raises an exception with the "abba" string as input. We’ll show how to use the Visual Studio debugger to figure out the problem with this simple function. The algorithm recursively tests the characters of the string pair-wise at the beginning and at the end of the string because a string is a palindrome if the first and last characters are equal and the substring obtained by removing them is a palindrome too. The s and e variables define the boundaries of the string to be tested and initially refer to the first and last characters of the input string. Recursion terminates when the outermost characters of the string to be tested differ or when you have tested the whole string and the indexes collide. Figure 18-1 shows the debugging session of the simple program. You set a breakpoint at the instruction that prints the result of the isPalindrome function for the "abba" string by clicking where the red circle is shown, which indicates the location of the breakpoint. When the program is started in debug mode, its execution stops at the breakpoint, and you can step through the statements. The current instruction is indicated by the yellow arrow, and the current statement is highlighted, as shown in Figure 18-1. Figure 18-1. The Visual Studio debugger The state of the program is accessible through a number of windows showing different aspects of the running program, usually docked at the bottom of the debugging window. It is possible, for instance, to inspect the state of the local variables of the current method (the Locals window showing the local variables and arguments, e and s in this example) or the state of the call stack to see the sequence of method calls (the Call Stack window). An important window is the Watch view, which can be used to write variable names and simple expressions and watch them change during the execution. It is also possible to evaluate expressions in the Immediate Syme_850-4C18.fm Page 526 Tuesday, October 16, 2007 12:43 PM 526 CHAPTER 18 ■ DEBUGGING AND TESTING F PROGRAMS window and invoke methods, as shown in Figure 18-1, where the simple expressions e and s are used. More views are available through the Debug menu, including the state of executing threads and the memory. In this simple example, you are examining why isPalindrome misbehaves for an input string of even length. As shown in Figure 18-1, the Watch window has been used to monitor the s and e variables intended to define the bounds of the substring that has still to be checked; in this case, the two indexes cross without ever becoming equal, which is the criteria used to successfully stop the recursion. This happens when s has value 2 and e has value 1 in the example. The symptom of the misbehavior of the function is that an exception is thrown; this is frequently where debugging starts. In this example, the exception would have been thrown a few steps forward when e gets value -1, which is an invalid index for accessing a character in a string. If we used stre as the watch expression or in the Immediate window, the problem would appear rather evident. Now that we have found the bug, we can fix it by extending the test from s = e to s = e to ensure that even if the end index becomes smaller than the starting one, we deal the situation appropriately. ■Note In Visual Studio and other Microsoft .NET debugging tools, the debugger expressions follow the C syntax, and arrays do not require the dot before the square braces. The most noticeable differences between C and F expression syntax are that access to arrays uses rather than . and the equality operator is == rather than =. Using Advanced Features of the Visual Studio Debugger We’ll now focus on relevant aspects of the debugging facilities that the CLR provides to managed applications and via tools such as the Visual Studio debugger. Consider the notion of breakpoint, an essential tool to mark a statement in the program where you want to suspend its execution and inspect the program state. Often a bug appears only under very specific conditions. Trivial bugs such as the one we have discussed are the easiest to track and the first to be fixed in a program. It can be difficult or even impossible to suspend program execution not at the first execution of a statement but only when certain conditions are satisfied. Many programmers introduce an if statement with a dummy state- ment for the body and set the breakpoint to the statement to suspend the program under the defined condition; this requires a recompilation of the program and a change to the source code, which may lead to further problems, particularly when several points of the program must be kept under control. A more effective strategy is to use conditional breakpoints, a powerful tool offered by the debugger. With a right-click on a breakpoint in the editor window or in the Breakpoints window (accessible through the Debug menu), a number of additional options become available. For each breakpoint it is possible to indicate the following: � A condition: An expression indicates a condition that must be satisfied by the program state in order to suspend the program execution. � A hit count: The number of times that the breakpoint should be hit before suspending the execution.Syme_850-4C18.fm Page 527 Tuesday, October 16, 2007 12:43 PM CHAPTER 18 ■ DEBUGGING AND TESTING F PROGRAMS 527 � A filter: A mechanism to filter the machine, process, and thread to select the set of threads that will be suspended when the breakpoint is hit. � An action: This is to be executed when the breakpoint is hit, causing the execution of a given action. Breakpoint conditions and hit counts are the most frequently used options. Hit count is useful when a bug appears only after a significant period of execution; for instance, when debug- ging a search engine, a bug may occur only after indexing gigabytes of data and the number of hits of the breakpoint can be determined. Conditional expressions are more useful when it is difficult to reproduce exactly the execution and when the number of times that the breakpoint is hit is variable. As for expressions typed in the Immediate window, conditional expressions are expressed as in C, and this is for all languages, since the debugger infrastructure within the CLR is designed to deal with compiled programs and ignores the source language. Sometimes it is necessary to debug a running program that has been started without the debugger; a typical situation is when debugging a service started through the Service snap-in of the Management Console or when debugging a Web application live that is executed by IIS rather than by the web server used for development by Visual Studio 2005. In these situations, it is possible to attach the debugger to a running process by selecting the Attach to Process item of the Debug menu and selecting the process to debug. There are standard processes that are generally known to programmers, such as w3p.exe, which is used by IIS to run application pools where ASP.NET applications run, or the svchost.exe process, which generally hosts Windows services. However, sometimes it can be difficult to find out which process is running the code to debug, since there are several of these generic process hosts for running applications. Debugging a program slows down significantly its speed since the debugger infrastructure injects code to monitor program execution. Conditional breakpoints tend to make the situation worse because every time the breakpoint is hit, the condition must be tested before resuming the standard execution. The CLR debugging infrastructure operates at the level of compiled assemblies; this has several implications. The objects and types that are visible to the debugger are those generated by the compiler and not always explicitly defined by the programmer in the source code; the program database information tends to preserve the mapping between the source and the compiled program, but sometimes the underlying structure surfaces to the user. On the other hand, it is possible to debug programs written in different programming languages, even when managed and unmanaged code must interoperate. ■Note One tricky problem with F programs can be debugging tail calls. We described tail calls in Chapter 8. In particular, when a tail call is executed, the calling stack frame will be removed prior to the call. This means the calls shown in the Visual Studio call stack window may not be complete. It may be missing entries that should, logically speaking, be present, according to the strict call sequence that caused a program to arrive at a particular point. Likewise, the debugger commands step-into and step-out can behave a little unusually when stepping into a tail call. Figure 18-2 shows a debugging session of the program discussed in Chapter 17; we have stepped into the HelloWorld method, which is a C function accessed through the PInvoke interface Syme_850-4C18.fm Page 528 Tuesday, October 16, 2007 12:43 PM 528 CHAPTER 18 ■ DEBUGGING AND TESTING F PROGRAMS as witnessed by the Call Stack window. To enable the cross-language debugging, we indicated in the project options, specifically in the Debug section, that the debugging scope is the whole program rather than the current project. Figure 18-2. Cross-language debugging with Visual Studio 2005 debugger Instrumenting Your Program with the System.Diagnostics Namespace A managed application can programmatically access the debugging services of the CLR through the types contained in the System.Diagnostics namespace. There are several types in the namespace encompassing several aspects of the runtime, including stack tracing, communi- cations with the debugger, performance counter access for reading statistics about the computer state (memory and CPU usage are typically available using them), and operating system processes handling. We’ll focus on the classes related to debugging and the debugger. There are mainly three ways to interact with the debugging infrastructure: �The Debug class is used to programmatically assert conditions in the program and output debugging and tracing information to debuggers and other listeners. �The Debugger class is used to interact with the debugger, check whether it is attached, and trigger breaks explicitly from the program. � The debugging attributes are a set of custom attributes that can be used to annotate the program to control its behavior (see Chapters 9 and 10 for more information about custom attributes).Syme_850-4C18.fm Page 529 Tuesday, October 16, 2007 12:43 PM CHAPTER 18 ■ DEBUGGING AND TESTING F PROGRAMS 529 The Debug class provides a mean to output diagnostics messages without assuming that the program has been compiled as a console application, and all the debug output is collected by one or more listeners that receive the output notifications and do something with them. Each listener is an instance of a class inherited from the TraceListener class and typically sends the output to the console, or to a file, or notifies the user with a dialog box (you can find more infor- mation about how to write a listener in the class library documentation). In the following example, we have instrumented the isPalindrome function with tracing statements: let isPalindrome (str:string) = let rec check(s:int, e:int) = Debug.WriteLine("check call") Debug.WriteLineIf((s = 0), "check: First call") Debug.Assert((s = 0 s str.Length), sprintf "s is out of bounds: %d" s) Debug.Assert((e = 0 e str.Length), sprintf "e is out of bounds: %d" e) if s = e s = e + 1 then true else if str.s str.e then false else check(s + 1, e - 1) The WriteXXX methods of the Debug class are used to output data of a running program and are a sophisticated version of the printf debugging approach, where the program is enriched with print statements that output useful information about its current state. In this case, however, it is possible to redirect all the messages to different media rather than just print them to the console. It is also possible to conditionally output messages to reduce the number of messages sent to the debug output. In the example, we output a message each time the check method is invoked and use the conditional output to mark the first invocation. Assertions are a well-known mechanism to assert conditions about the state of a running program, ensuring that at a given point in the program certain preconditions must hold; for instance, assertions are often used to ensure that the content of an option-valued variable is not None at some point in the program. During testing we want to ensure that if this precondition is not satisfied, the program execution will be suspended as soon as possible. This avoids a tracing back from the point where the undefined value of the variable would lead to an exception. The Assert method allows specifying a Boolean condition that must hold; otherwise, the given message is displayed, prompting the user with the failed assertion. Both debug output and assertions are statements that typically are useful during the program development, though when a release is made, these calls introduce unnecessary overhead. Often the program compiled with these extra checks is indicated as the checked version of the program. The .NET Framework designers devised a general mechanism to strip out the calls to methods under a particular condition with the help of the compiler. The ConditionalAttribute custom attribute is used to label methods whose calls are included in the program only if a given compilation symbol is defined; for the methods in the Debug type, it is the DEBUG symbol. The F compiler supports this mechanism, making it is possible to leverage these tools to instrument the F program in a way that is supported by the .NET infrastructure. The Debugger type lets you check whether the program is attached to a debugger and to trigger a break if required. It is also possible to programmatically launch the debugger using this type and send log messages to it. This type is used less often than the Debug type, but it may be useful if a bug arises only without an attached debugger. In this case, it is possible to program- matically start the debugging process when needed.Syme_850-4C18.fm Page 530 Tuesday, October 16, 2007 12:43 PM 530 CHAPTER 18 ■ DEBUGGING AND TESTING F PROGRAMS Another mechanism that allows controlling the interaction between a program and the debugger is based on a set of custom attributes in the System.Diagnostics namespace. Table 18-1 shows the attributes that control in part the behavior of the debugger. Table 18-1. Attributes Controlling Program Behavior Under Debug Attribute Description DebuggerBrowsableAttribute Determines whether and how a member is displayed in the debug window. DebuggerDisplayAttribute Indicates how a type or field should be displayed in the debug window. DebuggerHiddenAttribute The debugger may interpret this attribute and forbid interaction with the member annotated with it. DebuggerNonUserCodeAttribute Marks code that is not user written (for instance, designer-generated code) and that can be skipped to not complicate the debugging experience. DebuggerStepperBoundaryAttribute Used to locally override the usage of DebuggerNonUserCodeAttribute. DebuggerStepThroughAttribute The debugger may interpret this attribute and disallow stepping into the target method. DebuggerTypeProxyAttribute Indicates a type that is responsible for defining how a type is displayed in the debug window; it may affect debugging performance and should be used only when it is really necessary to radically change how a type is displayed. DebuggerVisualizerAttribute Indicates for a type the type that defines how to render it while debugging. These attributes allow you to control essentially two aspects of the debugging: how data is visualized by the debugger and how the debugger should behave with respect to the visibility of members. The ability to control how types are displayed by the debugger can be helpful to produce customized views of data that may significantly help inspect the program state into an aggre- gate view. The easiest way is to use the DebuggerDisplayAttribute attribute, which supports the customization of the text associated with a value in the debugger window; an object of that type can still be inspected in every field. Consider the following simple example: DebuggerDisplay("re+imi") type MyComplex= re : double im : double let c = re = 0.0; im = 0.0 Console.WriteLine("0+1i", c.re, c.im) Here we’re introducing a record named MyComplex with the classic definition of a complex number. The DebuggerDisplayAttribute attribute is used to annotate the type so that the debugger Syme_850-4C18.fm Page 531 Tuesday, October 16, 2007 12:43 PM CHAPTER 18 ■ DEBUGGING AND TESTING F PROGRAMS 531 will display its instances using the mathematical notation rather than just displaying the type name. The syntax allowed assumes that curly braces are used to indicate the name of a property whose value should be inserted in the format string. Figure 18-3 shows the result in the Visual Studio 2005 debugger: on the left side is how the debugger window appears when MyComplex is without the DebuggerDisplay annotation; on the right side the custom string appears, and the properties in the string appear in curly braces. As you can see, the difference is in the value field, and the structure can still be inspected. You can use a custom visualizer to fully customize the appearance of the data within the debugger, but it may affect debugging performance. Figure 18-3. The MyComplex type shown by the debugger without and with DebuggerDisplay Figure 18-3 is also interesting because it shows how the debugger displays information from the compiled program. In this case, the association between the name c and the runtime local variable has been lost, and the record appears because it has been compiled by the compiler as a pair of fields and public properties. The rest of the namespace contains classes to interact with the runtime: the event logging infrastructure, processes and threads management, and the representation of the stack of a thread. Stack manipulation can be useful if it is necessary to know the call sequence that leads to executing a particular method. The StackTrace type exposes a list of StackFrame objects that provide information about each method call on the stack. Debugging Concurrent and Graphical Applications Although a debugger is a fundamental tool for inspecting applications, it is not the Holy Grail, and it must be used carefully, being aware of the fact that the process will interfere with the normal execution of the application. The most relevant impact that the debugging process has over a running program is the influence over the execution timing, which is a critical aspect of concurrent and graphical applications, which are becoming common nowadays. Sometimes a bug even disappears while using the debugger because of these changes to execution timings. Debugging and testing concurrent applications can be particularly difficult because the use of a debugger is guaranteed to alter execution timings. There is no general rule for debugging concurrent applications, but here we briefly discuss how the debugger can be used in these cases. Consider this simple example of a multithreaded application: light open System open System.ThreadingSyme_850-4C18.fm Page 532 Tuesday, October 16, 2007 12:43 PM 532 CHAPTER 18 ■ DEBUGGING AND TESTING F PROGRAMS let t1 = Thread(fun () - while true do printf "Thread 1\n" ) let t2 = Thread(fun () - while true do printf "Thread 2\n" ) t1.Start() t2.Start() Threads t1 and t2 access the console, which is a shared resource; when we run the program without a debugger attached, the string printed by the two threads appears interleaved on the console. If you set a breakpoint on the two printf statements and you start a debugging session, you see that stepping automatically moves from one thread to the other and the output of the program is completely different from the one obtained without debugging; this is true also if you disable the breakpoints. The output is even more unbalanced if you set the breakpoint in only one of the two threads. We discussed shared memory multithreaded applications in Chapter 13. In these applica- tions, shared objects accessed by different threads are critical resources that may be viewed in the debugger. If the debug of a single thread fails, setting breakpoints in different threads may help to study the dynamic of the application, even if the full interaction of the threads cannot be fully simulated. If this approach fails, it may be useful to introduce tests inside the applica- tion and use the Debugger type only when a given condition occurs. Channel-based message- passing applications are generally easier to debug than those that rely on shared memory, because it is possible to monitor the communication end points using breakpoints or logging messages. Although the careful use of the debugger may help in debugging concurrent appli- cations, sometimes external observation is enough to influence a running program. In these cases, tracing through debug output becomes a viable alternative, and in fact, large systems have different levels of traces to monitor program execution while running. Graphical applications also present issues when debugging. As discussed in Chapter 11, the event loop of a GUI application is handled by a single thread, and if this is blocked, the GUI of the application will cease working until it is suspended in the debugger. Consider the following simple application: open System open System.Windows.Forms let f = new Form(Text="Hello world") let b = new Button(Text="Click me", Dock=DockStyle.Fill) b.Click.Add(fun _ - b.Text - "Click me again" MessageBox.Show("Hello world") ignore ) f.Controls.Add(b) f.Show() Application.Run(f)Syme_850-4C18.fm Page 533 Tuesday, October 16, 2007 12:43 PM CHAPTER 18 ■ DEBUGGING AND TESTING F PROGRAMS 533 If you set a breakpoint at the MessageBox statement and debug the application, then when the button is clicked, the debugger suspends execution, and the form stops responding. Moreover, the text of the button does not change until the execution is resumed. This effect is because the thread suspended by the debugger is responsible for handling GUI events, including the paint event that will refresh the content of the button updating the button label. More specifically, event handlers can affect the appearance of a form in two ways: by setting properties of graphical controls and by explicitly drawing using a Graphics object. In the first case, the change will not be noticed until the execution is resumed; this is because the property change usually asks for a refresh of the control appearance, which will eventually result in a paint event that must be processed by the thread that is suspended in the debugger. In the second case, updates are immediately visible when a statement involving drawing primitives is executed (unless double buffering has been enabled on the particular window). For example, consider the following program displaying a window with a number of vertical lines: open System open System.Windows.Forms open System.Drawing let f = new Form(Text="Hello world") f.Paint.Add(fun args - let g = args.Graphics for i = 0 to f.Width / 10 do g.DrawLine(Pens.Black, i10, 0, i10, f.Height) ) f.Show() Application.Run(f) You can set the breakpoint at the DrawLine statement and start debugging the application, paying attention to move the debugger window in order to make the application form visible. If you continue the execution one statement at a time, you can see the lines appearing in the form. In this case, the interaction with the graphical system does not trigger an event but inter- acts directly with the Graphics object by emitting graphic primitives that are rendered immediately. We have discussed the issues of debugging graphical applications by showing examples based on Windows Forms. The same considerations apply to all event systems where a thread is responsible for event notification. For graphical systems such as WPF based on the retention of graphic primitives, things work slightly differently, though analogous considerations can be made. Debugging and Testing with F Interactive Functional programming languages have traditionally addressed many debugging and testing issues through the ability to interactively evaluate statements of the program and print the value of variables, inspecting the program state interactively. F Interactive allows you to execute code fragments and quickly test them; moreover, the state of the FSI script can be inspected by querying values from the top level.Syme_850-4C18.fm Page 534 Tuesday, October 16, 2007 12:43 PM 534 CHAPTER 18 ■ DEBUGGING AND TESTING F PROGRAMS Development and testing using F Interactive can effectively reduce development time, because code fragments can be evaluated more than once without having to recompile the entire system. The Visual Studio add-in makes this process even more productive because code is edited in the development environment with type checking and IntelliSense, and code can be sent to F Interactive simply by selecting and pressing the Alt+Enter shortcut. In this scenario, the isPalindrome function from the previous section could have been developed incrementally and tested by simply invoking it with a test input argument. Once found and fixed, the function definition could have been evaluated again and tested for further bugs. During software development it is common practice to write simple programs to test specific features of software (we will discuss this topic more extensively in the “Unit Testing” section). With F Interactive, tests can be defined as functions stored into a file and selectively evaluated in Visual Studio. This approach can be useful in developing and defining new tests, but more specific tools can be used to run tests in a more organic way. Controlling F Interactive As you saw in Chapter 9, programs run within F Interactive have access to an object called fsi that lets you control some aspects of the interactive execution. This is contained in the assembly FSharp.Interactive.Settings.dll, which is automatically referenced in files ending .fsx and within F Interactive sessions. Table 18-2 shows some of the methods supported by this object. Table 18-2. Members on the fsi Object Member Type Description fsi.FloatingPointFormat string Gets or sets the format used for floating-point numbers, based on .NET Formatting specifications fsi.FormatProvider System.IFormatProvider Gets or sets the cultural format used for numbers, based on .NET Formatting specifications fsi.PrintWidth int Gets or sets the print width used for formatted text output fsi.PrintDepth int Gets or sets the depth of output for tree-structured data fsi.PrintLength int Gets or sets the length of output for lists and other linear data structures fsi.ShowProperties bool Gets or sets a flag indicating if properties should be printed for displayed values fsi.AddPrinter ('a - string) - unit Adds a printer for values compat- ible with the specific type 'aSyme_850-4C18.fm Page 535 Tuesday, October 16, 2007 12:43 PM CHAPTER 18 ■ DEBUGGING AND TESTING F PROGRAMS 535 Table 18-2. Members on the fsi Object Member Type Description fsi.AddPrintTransformer ('a - obj) - unit Adds a printer that shows any values compatible with the specific type 'a as if they were values returned by the given function fsi.CommandLineArgs string Gets the command-line arguments after ignoring the arguments rele- vant to the interactive environment and replacing the first argument with the name of the last script file Some Common F Interactive Directives Table 18-3 shows some common directives accepted by F Interactive, some of which corre- spond to options for the F command-line compiler. Table 18-3. Some Commonly Used F Interactive Directives Directive Description r path References a DLL. The DLL will be loaded dynamically when first required. I path Adds the given search path to that used to resolve referenced DLLs. use file Accepts input from the given file. load file ... file Loads the given file(s) as if it had been compiled by the F command- line compiler. time Toggles timing information on/off. quit Exits F Interactive. Understanding How F Interactive Compiles Code Although the F Interactive is reminiscent of the read-eval-print loops of interpreted languages, it is substantially different because it compiles code rather than interprets it. Whenever a code fragment is typed on the top level, it gets compiled on the fly as part of a dynamic assembly, and it gets evaluated for side effects. This is particularly important for types because it is possible to create new ones at the top level and their dependencies may be tricky to fully understand. We start with an example of nontrivial use of F Interactive that shows these intricacies, and we define the class APoint representing points using an angle and a radius: type APoint(angle,radius) = member x.Angle = angle member x.Radius = radius new() = APoint(angle=0.0, radius=0.0)Syme_850-4C18.fm Page 536 Tuesday, October 16, 2007 12:43 PM 536 CHAPTER 18 ■ DEBUGGING AND TESTING F PROGRAMS If you create an instance of the class using F Interactive, you can inspect the actual type by using the GetType method, and you get the following output: let p = APoint();; val p : APoint p.GetType();; val it : System.Type = FSI_0002+APoint Assembly = FSI-ASSEMBLY, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null; AssemblyQualifiedName = "FSI_0002+APoint, FSI-ASSEMBLY, Version=0.0.0.0, ... Now suppose you want to extend the APoint class with an additional member that stretches the point radius of a given amount; it is natural to type the new definition of the class into the top level and evaluate it. And in fact F Interactive does not complain about the redefinition of the type with the following: type APoint(angle,radius) = member x.Angle = angle member x.Radius = radius member x.Stretch (k:double) = APoint(angle=x.Angle, radius=x.Radius + k) new() = APoint(angle=0.0, radius=0.0) Since we have redefined the structure of APoint, we may be tempted to invoke the stretch method on it, but we get an error: p.Stretch(22.0);; p.Stretch(22.0);; stdin(2,2): error: FS0039: The field, constructor or member 'Stretch' is not defined. To really understand what is happening, we create a new instance p2 of the class APoint and ask for the type: let p2 = APoint();; val p2 : APoint p2.GetType();; val it : System.Type = FSI_0005+APoint Assembly = FSI-ASSEMBLY, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null; AssemblyQualifiedName = "FSI_0005+APoint, FSI-ASSEMBLY, Version=0.0.0.0, ... Syme_850-4C18.fm Page 537 Tuesday, October 16, 2007 12:43 PM CHAPTER 18 ■ DEBUGGING AND TESTING F PROGRAMS 537 As you can see, the name of the type of p2 is FSI_0005+APoint, whereas the type of p is FSI_0002+APoint and is different. Under the hood, F Interactive compiles types into different modules to ensure that types can be redefined and ensures that the most recent definition of a type is used. Besides, the older definitions are still available, and their instances are not affected by the type redefinition. Understanding the inner mechanisms of F Interactive is useful when using it to test F programs because interactive evaluation is not always equivalent to running code compiled using the command-line compiler. On the other hand, the compiled nature of the system guar- antees that the code executed by F Interactive performs as well as compiled code. Unit Testing Software testing is an important task in software development; its goal is to ensure that a program or a library behaves according to the system specifications. It is a relevant area of software engineering research, and tools have been developed to support the increasing effort of soft- ware verification. Among a large number of testing strategies, unit testing has become rapidly popular because of software tools used to support this strategy. The core idea behind this approach is that programmers often write small programs to test single features of a system during devel- opment. When bugs are found, new unit tests are added to ensure that a particular bug does not occur again. Recently it has been proposed that testing should drive software development, because tests can be used while developing programs to check new code and later to conduct regression tests, ensuring that new features do not affect existing ones. In this section, we will discuss how test units can be developed in F using the NUnit tool from http://www.nunit.org, a freely available tool supporting this testing strategy. The tool was inspired from JUnit, a unit testing suite for the Java programming language, but the inter- face has been redesigned to take advantage of the extensible metadata provided by the CLR by means of custom attributes. To make the experience more concrete, we’ll start with an example and develop a very simple test suite for the isPalindrome function. The first choice you have to face is whether tests should be embedded into the application. If tests are created as a separated application, you can invoke only the public interface of your software, and features internal to the software cannot be tested directly; on the other hand, if you embed unit tests within the program, you introduce a dependency from the nunit.framework.dll assembly, and unit tests are available at runtime even where unneeded. Because the NUnit approach is based on custom attributes, performance is not affected in both cases. If tests are used during program development, it is more convenient to define them inside the program; in this case, conditional compilation may help to include them only in checked builds. Listing 18-1 shows a test fixture for the isPalindrome function, which is a set of unit tests. Test fixtures are represented by a class annotated with the TestFixture custom attribute, and tests are instance methods with the signature unit - unit and annotated with the Test custom attribute. Inside a test case, methods of the Assert class are used for testing conditions that have to be satisfied during the test. If one of these fails, the test is considered failed, and it is reported to the user by the tool that coordinates test execution.Syme_850-4C18.fm Page 538 Tuesday, October 16, 2007 12:43 PM 538 CHAPTER 18 ■ DEBUGGING AND TESTING F PROGRAMS Listing 18-1. A Test Fixture for the isPalindrome Function light open System open NUnit.Framework open IsPalindrome TestFixture type Test() = let posTests(strings) = for s in strings do Assert.IsTrue(isPalindrome s, sprintf "isPalindrome(\"%s\") must return true" s) let negTests(strings) = for s in strings do Assert.IsFalse(isPalindrome s, sprintf "isPalindrome(\"%s\") must return false" s) Test member x.EmptyString () = Assert.IsTrue(isPalindrome(""), "isPalindrome must return true on an empty string") Test member x.SingleChar () = posTests "a" Test member x.EvenPalindrome () = posTests "aa"; "abba"; "abaaba" Test member x.OddPalindrome () = posTests "aba"; "abbba"; "abababa" Test member x.WrongString () = negTests "as"; "F is wonderful"; "Nice" Test units are simply methods that invoke objects of the program and test return values to check that its behavior conforms to the specification. As you can see, we also introduced the posTests and negTests functions used in several tests. Developing unit tests is simply a matter of defining types containing the tests. Although it is possible to write a single test for a program, it is a good idea to have many small tests checking various features and different inputs. In this case, we have been able to introduce five tests for a simple function; of course, we did it to show the main idea, but nevertheless we have developed a test for each significant input to the func- tion. We could have developed a single test with all the code used for the single tests together, but as you will see shortly, this would have reduced the ability of the test suite to spot problems in the program. In general, the choice of the granularity of a test suite for a program is up to the Syme_850-4C18.fm Page 539 Tuesday, October 16, 2007 12:43 PM CHAPTER 18 ■ DEBUGGING AND TESTING F PROGRAMS 539 developer, and it is a matter of finding a reasonable trade-off between having a large number of unit tests checking very specific conditions and having a small number of unit tests checking broader areas of the program. To compile the project, you must reference the nunit.framework.dll assembly; usually the -R compiler switch is used to ensure that the assembly is copied in the output directory of the program. Once the program has been compiled, you can start NUnit and open the executable. As shown in Figure 18-4, the assembly containing the unit tests has been inspected using the reflection capabilities of the CLR, the classes annotated with the TestFixture attribute are identified by NUnit, and searched-for methods are annotated with the Test attribute. Initially, all the fixtures and the tests are marked with gray dots. When tests are run, the dot is colored green or red depending on the outcome of the particular test. Figure 18-4. Test unit of isPalindrome executed in NUnit If we reintroduce the original bug in the isPalindrome function and run NUnit again, EmptyString and EvenPalindrome will fail, the corresponding dots will be marked as red, and the Errors and Failures tab will contain details about the test failure. This is the main benefit of having a large number of small unit tests: tools may run them automatically and help identify problems in a program as well as the area potentially involved in the problem. Even in this simple example, a single test for the whole function would have indicated the problem with the function but failed to spot the kind of input responsible for the problem. As every other piece of software, unit tests have to be maintained, documented, and updated to follow the evolution of the software for which they are designed. The number of test cases, organized in fixtures, tends to grow with the system during development, and in a large system it is possible to have thousands of these tests. Tools such as NUnit have features to control tests and allow you to run subsets of the whole set of test cases for a system. The notion of test fixtures Syme_850-4C18.fm Page 540 Tuesday, October 16, 2007 12:43 PM 540 CHAPTER 18 ■ DEBUGGING AND TESTING F PROGRAMS that you have already encountered is a form of grouping: a test suite may contain different test fixtures that may group test cases for different aspects to be tested. NUnit features a number of additional attributes to support the documentation and classifica- tion of test cases and test fixtures. The Description attribute allows associating a description with annotated test fixtures and test cases. Category and Culture attributes can be used to associate a category and a culture string with test fixtures and test cases; in addition, to provide more information about tests, NUnit allows filtering tests to be run using the content of the attributes. The ability to select the tests that must be run is important because running all tests for a system may require a significant amount of time. Other mechanisms to control the execution of tests are offered by the Ignore and Explicit attributes; the former can be used to disable a test fixture for a period without having to remove all the annotations, and the latter indicates that a test case or a fixture should be run only explicitly. Another important area for testing nontrivial software is the life cycle of a test fixture. Test cases are instance methods of a class, and with a simple experiment, you can easily find that NUnit creates an instance of the class and runs all the tests it contains. To verify this, it is enough to define a counter field in the class annotated as a fixture and update its value every time a test is run; the value of the counter is consistently incremented for each test in the suite. Although you may relay on the standard life cycle of the class, NUnit provides additional annotations to indicate the code that must be run to set up a fixture and the corresponding code to free the resources at the end of the test; it is also possible to define a pair of methods that are run before and after each test case. The attributes controlling these aspects are TestFixtureSetUp and TestFixtureTearDown for annotating methods to set up and free a fixture and SetUp and TearDown for the corresponding test cases. Listing 18-2. A Refined Test Fixture for the isPalindrome Function light open System open NUnit.Framework open Debug TestFixture; Description("Test fixture for the isPalindrome function") type Test() = TestFixtureSetUp member x.InitTestFixture () = printfn "Before running Fixture" TestFixtureTearDown member x.DoneTestFixture () = printfn "After running Fixture" SetUp member x.InitTest () = printfn "Before running test"Syme_850-4C18.fm Page 541 Tuesday, October 16, 2007 12:43 PM CHAPTER 18 ■ DEBUGGING AND TESTING F PROGRAMS 541 TearDown member x.DoneTest () = Console.WriteLine("After running test") Test; Category("Special case"); Description("An empty string is palindrome") member x.EmptyString () = Assert.IsTrue(isPalindrome(""), "isPalindrome must return true on an empty string") Listing 18-2 shows a test fixture for the isPalindrome function that includes most of the attributes we have discussed and one test case. We mark the category of this test case as a “Special case.” We also include a description for each test case and the methods invoked before and after the fixture and single test cases are run. The graphical interface of NUnit provides a tab reporting the output sent to the console, and when tests run, the output shows the invoca- tion sequence of the setup and teardown methods. The ability to set up resources for test cases may introduce problems in the unit testing; in particular, the setup and teardown methods of test fixtures must be treated carefully because the state shared by different test cases may affect the way they execute. Suppose, for instance, that a file is open during the setup of a fixture. This may save time because the file is opened only once and not for each test case. If a test case fails and the file is closed, the subsequent tests may fail because they assume that the file has been opened during the setup of the fixture. Nevertheless, there may be situations where the ability of preloading resources only once for a fixture can save a significant amount of time. NUnit comes with two versions of the tool: one displaying the graphical interface shown in Figure 18-4 and a console version of the tool printing the results to the console. Both versions are useful; the windowed application is handy to produce reports about tests and interactively control the test processing, and the console version can be used to include the test process into a chain of commands invoked via scripts. Also, the output of the tool can be read by other programs to automate tasks after unit tests. A large number of command-line arguments are available in the console version to specify all the options available, including test filtering based on categories. When a unit test fails, there is the problem of setting up a debugging session to check the application state and the reason of the failure. It is possible to debug tests using the Visual Studio debugger by simply configuring the Debug tab in the project properties in a similar way, as shown in Figure 18-5. Once configured, it is possible to set breakpoints in the code and start the debugging session as usual. This is important when code development is driven by tests, since new features can be implemented alongside test cases. This is a good way to capitalize on the small test programs that developers frequently write. These small programs become the test cases and can be collected without having to develop a new test program each time. In the example shown in Figure 18-5, we pass a single argument to nunit-console.exe, the assembly containing the tests to be executed. It is also possible to specify an additional argument to filter the tests that must be run. In this example, if you set a breakpoint in one of the test cases annotated explicitly, the debugger will not stop, because by default these tests are skipped.Syme_850-4C18.fm Page 542 Tuesday, October 16, 2007 12:43 PM 542 CHAPTER 18 ■ DEBUGGING AND TESTING F PROGRAMS Figure 18-5. Configuring an F project for debugging NUnit test fixtures ■Note In this section we have shown how you can use NUnit to define test cases using F. NUnit is not the only tool for unit testing available for .NET. For example, Visual Studio 2005 includes powerful unit testing tools. NUnit also integrates with other software development tools such as NCover, a coverage testing tool capable of conducting coverage tests driven by test fixtures and test cases.