Generic Class Factory – C# Part 4 / 4

This is the last post in the generic class factory series.  The previous posts included:

• Generic factory introduction.  Please read this post first to understand why a class factory should be used instead of simply creating an object and how a generic class factory works.   This introductory post introduces a C++ generic class factory that only works to create objects of the same base class.
• C++ generic factory that can be used to create objects of any type.  Most, if not all generic factory examples previously published only work to create objects of a specific base class or concrete type.  This post shows how to design a class factory that can be used to create any type of object – even those with different base types.  I have not seen this technique published before.
• Elegant C# generic factory.  This post shows how to use the Type class and the reflection Activator object to link up to the proper constructor without having to define static creation methods required by the C++ generic factory classes.  The drawback of this technique is that it is significantly slower than creating an object with the new keyword.

In the elegant C# generic factory post, a benchmark was provided that shows the generic class factory is over 23 times slower than creating an object with the new keyword.  It was suggested that this is unlikely to significantly affect overall performance if objects are infrequently created.  It was also pointed out that objects that are frequently created and discarded, that performance could be improved by using an object pool for discarded objects and then simply retrieving, removing the object from the pool, and reinitializing the object as needed.  This should also help to minimize the impact of the garbage collector.

If an object pool can’t be used for some reason, and high performance of creation of objects from a generic factory is critical, then this post should help shed some light on how to achieve this objective.

This C# class uses the same model as used in the C++ generic classes, but uses a delegate instead of a function pointer in order to avoid the performance issues with reflection.  Delegates are somewhat similar to a C++ function pointer, but are type safe and can be used as callback methods.  Here is the full class:

A C# Generic Class Designed For Performance

public class GenericFactoryDelg<TKey, TType>
{
   // This class can be used to create an object of any type.
   //
   public delegate TType CreateObjectDelegate(params object[] args);

   private readonly Dictionary<TKey, CreateObjectDelegate> FactoryMap; // Used to map the key to the delegate static creation method used to create the object.

   public GenericFactoryDelg()
   {
      FactoryMap = new Dictionary<TKey, CreateObjectDelegate>();
   }

   public void Register(TKey key, CreateObjectDelegate ctor)
   {
      // This method registers a delegate to the method used to create the object so that it can be created later.
      //
      FactoryMap.Add(key, ctor);
   }

   public DType CreateObj<DType>(TKey id, params object[] args) where DType : TType
   {
      // This method creates an object based upon the specified key that was used to register the type earlier.
      // The Register method must have been called for this key, or an exception will be thrown.
      //
      CreateObjectDelegate Constructor = null;
      // Get the delegate to the constructor used to create the object, and then call it.
      if (FactoryMap.TryGetValue(id, out Constructor))
         return (DType)Constructor(args);
      throw new ArgumentException("No type registered for this id.");
   }
}

The delegate CreateObjectDelegate is defined as:

public delegate TType CreateObjectDelegate(params object[] args);

Like the C++ code, TType is a generic class parameter that identifies the type of the class to be created and the consumer of this class is responsible for defining a static creation method to create the object.  The FactoryMap dictionary object is used in the Register method to keep track of the delegate that is called in CreateObj.  In CreateObj, the generic constraint “where DType : TType” is required for the code to compile.  If the constraint is removed, then the compiler error “Cannot convert type ‘TType’ to ‘DType'” is produced.

This generic class factory can be used to create objects of unrelated types if object is used for the TTYPE class parameter.  For example:

GenericFactoryDelg<int, object> GFD = new GenericFactoryDelg<int, object>();

A delegate to the method used to create an object must be defined to use the CreateObjectDelegate defined in GenericFactoryDelg.  For example:

GenericFactoryDelg<int, object>.CreateObjectDelegate CreateRedColor = new GenericFactoryDelg<int, object>.CreateObjectDelegate(RedColor.CreateObj);

The Register method must be called to associate the creation delegate with the key.  For example:

GFD.Register(1, CreateRedColor);

The object can be created by calling the CreateObj method with the appropriate key.  For example:

Color Red3 = GFF.CreateObj<Color>(1, 3, "Name3");

The following code tests the GenericFactoryDelg class and verifies that the objects were successfully created:

// Create a generic factory that works with int keys.
GenericFactoryDelg<int, object> GFD = new GenericFactoryDelg<int, object>();
// Create the delegates used to create the objects.
GenericFactoryDelg<int, object>.CreateObjectDelegate CreateRedColor = new GenericFactoryDelg<int, object>.CreateObjectDelegate(RedColor.CreateObj);
GenericFactoryDelg<int, object>.CreateObjectDelegate CreateBlueColor = new GenericFactoryDelg<int, object>.CreateObjectDelegate(BlueColor.CreateObj);
GenericFactoryDelg<int, object>.CreateObjectDelegate CreateDogAnimal = new GenericFactoryDelg<int, object>.CreateObjectDelegate(Dog.CreateObj);
GenericFactoryDelg<int, object>.CreateObjectDelegate CreateHorseAnimal = new GenericFactoryDelg<int, object>.CreateObjectDelegate(Horse.CreateObj);
// Register each type that can be created with the generic class factory.
GFD.Register(1, CreateRedColor);
GFD.Register(2, CreateBlueColor);
GFD.Register(3, CreateDogAnimal);
GFD.Register(4, CreateHorseAnimal);
//
// Create a Red color object into an object and convert it to a Color object.
object ColorObj = GFD.CreateObj<object>(1);
Color Red1 = (Color)ColorObj;
if (Red1.Hue != 0)
   Console.WriteLine("TestDelegateGenericFactory:GF2.CreateObj() did not call the proper RedColor(0) constructor.");
// Create a Blue object and save it into a Color base class object.
Color Blue2 = GFD.CreateObj<Color>(2, 2);
if (Blue2.Hue != 2)
   Console.WriteLine("TestDelegateGenericFactory:GF2.CreateObj() did not call the proper BlueColor(2) constructor.");
// Create a Red object.
RedColor Red3 = GFD.CreateObj<RedColor>(1, 3, "Name3");
if (Red3.Hue != 3)
   Console.WriteLine("TestDelegateGenericFactory:GF2.CreateObj() did not call the proper RedColor(3) constructor.");
// Using the same generic factory, create a completely different object that has nothing to do with colors.
object AnimalObj = GFD.CreateObj<object>(3);
Animal Pup1 = (Animal)AnimalObj;
if (Pup1.YearsOld != 0)
   Console.WriteLine("TestDelegateGenericFactory:GF2.CreateObj() did not call the proper Dog(0) constructor.");
// Create a Horse object and save it into an Animal base class object.
Animal Horse2 = GFD.CreateObj<Animal>(4, 2);
if (Horse2.YearsOld != 2)
   Console.WriteLine("TestDelegateGenericFactory:GF2.CreateObj() did not call the proper Horse(2) constructor.");
// Create another dog object.
Dog Pup2 = GFD.CreateObj<Dog>(3, 4, "Rex");
if ((Pup2.YearsOld != 4) && (Pup2.Name != "Rex"))
   Console.WriteLine("TestDelegateGenericFactory:GF2.CreateObj() did not call the proper Dog(4) constructor.");

The performance of this generic class factory is much improved over the one that uses the Activator class; it is over six times faster.  See Class Factory Benchmark Results.txt in the download file for more information.

Series Conclusion

These series of posts have shown how a generic C++ class factory can be implemented to work with unrelated types.  An elegant C# generic class factory has been presented.  The benefit of this class is that it directly calls a class’s constructor and does not require static creation methods like the C++ generic class factory – which should save some code and time.  The drawback to this elegant class is that it is a little slow since it uses reflection to hook up to the proper constructor.  Finally, A C# generic class factory was also presented that is over 6 times faster.  The drawback to this class is that static creation methods must be implemented like the C++ generic class factory.

The code for this class may be downloaded by following this link.

Generic Class Factory – C# Part 3 / 4

This 3rd post in the generic class factory series focuses on creating objects in C# with an elegant class that uses the Type class and reflection’s Activator object.  The first post in the series provides an introduction to the generic class factory pattern, when it should be used, how it works, and shows a simple implementation in C++.  If you are not familiar with the generic class factory pattern, then please read this post first.  The second post is also in C++ and shows how a generic class factory can be designed to create unrelated types.

The following C# generic class factory uses the Type class.  The Type class is the root of System.Reflection and provides a way to access metadata.  Types can be passed to methods without using generics, stored in a container, and then retrieved at a later time to create an object.  There is no way to store a type in a container in unmanaged C++ and no way to create an object from a stored type.  What this means in C# is that the constructor for an object can be called directly by the CreateObj method – which saves code and time by not having to implement the static creation methods.  Here is the full class:

An Elegant C# Generic Class For Creating Unrelated Objects

public class GenericFactory<TKey>
{
   //
   private readonly Dictionary<TKey, Type> FactoryMap; // Used to map the key to the type of object.
   //
   public GenericFactory()
   {
      // Create the dicionary that will hold the key and type of object associated with that key.
      FactoryMap = new Dictionary<TKey, Type>();
   }

   public void Register(TKey key, Type thisType)
   {
      // This method registers a type so that it can be created later.
      // An exception is thrown if the type to be registered is either an interface or abstract.
      //
      if ((thisType.IsInterface) || (thisType.IsAbstract))
         throw new ArgumentException("GenericFactory::Register: type must not be an interface or abstract.");
      FactoryMap.Add(key, thisType);
   }

   public TType CreateObj<TType>(TKey key, params object[] args)
   {
      // This method creates an object based upon the specified key that was used to register the type earlier.
      // The Register method must have been called for this key, or an exception will be thrown.
      //
      Type ThisType;
      if (FactoryMap.TryGetValue(key, out ThisType))
         return (TType)Activator.CreateInstance(ThisType, args);
      // The key was not found.  Most likely the key was not properly registered.  So throw an exception here.
      throw new ArgumentException("GenericFactory::CreateObj: type has not been registered for this key.");
   }
}

The Register and CreateObj methods are used to manage objects of various types.  The C++ function pointer has been replaced by the Type class in C#, which is stored in the dictionary.  What this implies is that the static object creation methods that had to be written in C++ don’t have to be written in C# – since the constructors can be called directly by the class factory.  This is a real advantage that C# has over unmanaged C++.  To accomplish this little feat, the .NET Framework library offers an Activator class that can be used to create any type with any number of arguments, or even no arguments.  C++ does not offer anything like the Activator class.

In the CreateObj method, an array of objects replace the vector of void pointers defined in the C++ generic classes.  The params keyword is used so that a variable number of individual parameters can be specified or an array of objects used instead.  There must be a constructor defined that accepts the specified number of parameters.

The CreateObj method, which is defined as:

public TType CreateObj<TType>(TKey key, params object[] args)

calls the CreateInstance method on the Activator object:

return (TType)Activator.CreateInstance(ThisType, args);

The constructor for the type that gets called depends on the number of elements in args.  If args is null, then the default parameterless constructor is called.  If args has just one element, then the constructor with one parameter is called.  If args has two elements, then then the constructor with two parameters is called.  This provides quite a bit of flexibility to control the constructor that is called.  To see this in action, try debugging through the code and stepping into the CreateInstance method.  Three constructors are provided in the test classes, which are defined in the TestHelperClasses.cs (located in the download file), and the test code below tests each of them.

The following code tests the C# generic factory that uses the Type class to manage object creation and verifies that all the objects were created correctly:

// Create a generic factory that works with key strings.
GenericFactory<string> GenFact = new GenericFactory<string>();
// Register each type of object that can be created.
GenFact.Register("Red", typeof(RedColor));
GenFact.Register("Blue", typeof(BlueColor));
GenFact.Register("Pup", typeof(Dog));
GenFact.Register("Nag", typeof(Horse));
// Create a Red color object and save it into an object.
object ColorObj = GenFact.CreateObj<object>("Red");
Color Red1 = (Color)ColorObj;
if (Red1.Hue != 0)
   Console.WriteLine("TestTypeGenericFactory:GenFact.CreateObj() did not call the proper RedColor(0) constructor.");
// Create a Blue object and save it into a Color base class object.
Color Blue2 = GenFact.CreateObj<Color>("Blue", 2);
if (Blue2.Hue != 2)
   Console.WriteLine("TestTypeGenericFactory:GenFact.CreateObj() did not call the proper BlueColor(2) constructor.");
// Create a RedColor object.
RedColor Red3 = GenFact.CreateObj<RedColor>("Red", 3, "Name3");
if ((Red3.Hue != 3) && (Red3.Name != "Name3"))
   Console.WriteLine("TestTypeGenericFactory:GenFact.CreateObj() did not call the proper RedColor(3) constructor.");
// Using the same generic factory, create a completely different object that has nothing to do with colors.
object AnimalObj = GenFact.CreateObj<object>("Pup");
Animal Pup1 = (Animal)AnimalObj;
if (Pup1.YearsOld != 0)
   Console.WriteLine("TestTypeGenericFactory:GenFact.CreateObj() did not call the proper Dog(0) constructor.");
// Create a Horse object and save it into an Animal base class object.
Animal Horse2 = GenFact.CreateObj<Horse>("Nag", 2);
if (Horse2.YearsOld != 2)
   Console.WriteLine("TestTypeGenericFactory:GenFact.CreateObj() did not call the proper Horse(2) constructor.");
// Create another Dog object.
Dog Pup3 = GenFact.CreateObj<Dog>("Pup", 3, "Rex");
if ((Pup3.YearsOld != 3) && (Pup3.Name != "Rex"))
   Console.WriteLine("TestTypeGenericFactory:GenFact.CreateObj() did not call the proper Dog(3) constructor.");

Performance Concerns

When working with reflection, it is natural to be a little concerned about performance.  The Activator class provides some very nice functionality, but at what price performance wise?  The results of benchmark code that compares how fast an object can be created with the Activator class with how fast it can be created by using the new operator is summarized below (see Class Factory Benchmark Results.txt in the download file for full results).  The two argument constructor is called in both cases:

Activator.CreateInstance took 185.6ms to create 100,000 objects, or about .0019ms per object created.  Whereas using new took just 8ms to create 100,000 objects, which works out to be .00008ms per object created.  That corresponds to a 185.6 / 8 = 23.2 times longer to create an object with the Activator class compared to using the new operator.

Most of the time, 1/500th of a millisecond to construct an object should not be a major concern for most applications – especially if objects are created somewhat infrequently.  However, in the case where performance does matter, then an object pool could be used to store objects that are no longer being used.  One example of this is where objects are frequently created, used for a short time, and then discarded.  In this situation, an object pool could be used to hold the discarded objects.  If the discarded object pool is empty, than a new object is created.  Otherwise, an object would be retrieved and removed from the object pool.  When the object is no longer needed, it would be added to the object pool.  Another benefit to this approach is that the garbage collector would not need to work as hard.

An initialization method would need to be written that matches the constructor so that the object can be properly initialized with new parameter data.  It would be more modular to have the constructor call the initialization method in this case so that the object creation code can be centralized.  As object construction becomes more complex and time consuming, this idea can be extended by leaving parts of an object intact and only re-initializing the parts of an object that need specific reconstruction – which of course depends on the requirements of the application.

If greater object creation performance is desired, then one last C# generic class is provided, which is the subject of the last post in this series on generic class factories.

The code for this class may be downloaded by following this link.

Generic Class Factory – C++ Part 2 / 4

This 2nd post in the generic class factory series focuses on creating objects of different types in C++, which I have not seen published before.  This means that just one generic class factory can create objects that have no common base class – which will save on code and complexity when having to deal with many different base class objects that need to be created.

The first post in the series introduced a generic class factory that can only work to create objects of the same base class, when a class factory should be used instead of simply creating an object, and how a generic class factory works.

This is the last post in the series for C++.  The last two posts will deal with C# generic class factories.

A  C++ Generic Class To Create Unrelated Objects

template <class TKey>
class GenericFactories
{
   // This class implements a generic factory that can be used to create any type.
   typedef void* (*CreateObjFn)(vector<void*> &args); // Defines the CreateObjFn function pointer that points to the object creation function.
   typedef tr1::unordered_map<TKey, CreateObjFn> FactoryMap; // Hash table to map the key to the function used to create the object.
public:
   void Register(const TKey &keyName, CreateObjFn pCreateFn)
   {
      // Store the object type name in the hash table.
      FactMap[keyName] = pCreateFn;
   }

   template <class TType>
   TType* CreateObj(const TKey &keyName, vector<void*> &args)
   {
      // This method looks for the name in the hash map.  If it is not found, then an exception is thrown.
      // If it is found, then it creates the specified object returns a pointer to it.
      //
      FactoryMap::iterator It = FactMap.find(keyName);
      if (It != FactMap.end())
      {
         void* pObj = It->second(args); // Call the method to create the object.
         TType* Obj = static_cast<TType*>(pObj); // Cast the object to the specified type.
         return Obj;
      }
      throw "GenericFactories::CreateObj: key was not found in hashtable.  Did you forget to register it?";
   }
private:
   FactoryMap FactMap;
};

TKey is the only class parameter, which is used as the type of key.  This is used to keep track of the type of object to be created.  The typedef:

typedef void* (*CreateObjFn)(vector<void*> &args);

is subtlety different from the typedef in the first generic class factory described in the first post.  It now returns a void pointer instead of the base class parameter type.  This is one of the insights required to make it work with unrelated types.  But, this also means that the static creation methods used to create the object must be defined to return a null pointer.  As in the first post, the user is required to create the static creation method to create the object.

The Register method must be called prior to calling CreateObj so that the generic factory knows which method to call to create the object.  As in the first post, a hashtable is used to keep track of the key and the static creation method.

The new CreateObj method introduces a new function template parameter, which is called TType.  By moving the class template parameter out of the class definition to a function template parameter in the CreateObj method means that the class is no longer constrained to creating objects of a specific base class.  Any type of class can now be created by this factory – even classes with very different base types.

When creating the object, the base class family must be supplied to CreateObj as the function template parameter in order for the correct type to be returned.  A static_cast is done to convert the void pointer returned by the static creation object method to the expected type.

The following code shows how to use this generic factory:

GenericFactories<int> GenFacts; // Create a generic class factory that works with int keys.
GenFacts.Register(1, &RedColor::CreateObj); // Register 1 to create a red color object.
GenFacts.Register(100, &Dog::CreateObj); // Register 100 to create a dog object.
// Prepare the arguments needed to create a red object with a hue of 15.
vector<void*> Args;
Args.push_back("Light Red");
int Hue = 15;
Args.push_back(&Hue);
// Create a red color object from the factory.
unique_ptr<Color> pRedColor(GenFacts.CreateObj<Color>(1, Args));
cout << "Color object is of type " << pRedColor->GetType() << endl;
// Now create a completely different object from the factory - a dog animal object.
Args[0] = "Max";
int YearsOld = 4;
Args[1] = &YearsOld;
unique_ptr<Animal> pDoggy(GenFacts.CreateObj<Animal>(100, Args));
cout << "Animal object is of type " << pDoggy->GetType() << endl;

The code for this class may be downloaded by following this link.

Benchmark of foreach .vs. for loop

The foreach loop provides a nice way to iterate through a generic container.  But, is there a price to pay for it performance wise .vs. using a for loop? It turns out that there is a performance hit when using a foreach loop.

The following benchmark code was used to determine how well a foreach loop compares against a for loop. The code was run 5 times in release mode, and the average taken:

Time Elapsed for the for loop = 5 ms
Time Elapsed for the for next loop = 12 ms – press a key to continue.

Time Elapsed for the for loop = 6 ms
Time Elapsed for the for next loop = 12 ms – press a key to continue.

Time Elapsed for the for loop = 6 ms
Time Elapsed for the for next loop = 12 ms – press a key to continue.

Time Elapsed for the for loop = 5 ms
Time Elapsed for the for next loop = 12 ms – press a key to continue.

Time Elapsed for the for loop = 6 ms
Time Elapsed for the for next loop = 12 ms – press a key to continue.

Total for the for loop = 33ms. Avg = 28 / 5 = 5.6ms for 2M iterations.
Total for the foreach loop = 60ms. Avg = 60 / 5 = 12ms for 2M iterations.

Average ratio = 12 / 5.6 = 2.14.

RESULT: The for loop is a little more than twice as fast as the foreach loop.

The code is given below:

public abstract class Color
{
    public abstract int DoMethod();
}

class RedColor : Color
{
    private int Value;
    public int TestValue
    {
        get { return Value; }
        set { Value = value; }
    }

    public RedColor()
    {
        Value = 0;
    }

    public RedColor(int value)
    {
        Value = value;
    }

    public override int DoMethod()
    {
        return Value;
    }
 }

public static void BenchmarkFor()
{
    // This method benchmarks the relative performance of using a for next loop .vs. a foreach loop.
    //
    const int Iters = 2000000;
    int Loop;
    // NOTE - RedColor is a very simple class. It can really be anything. 
    List<RedColor> GenList = new List<RedColor>(Iters);
    RedColor RC;
    GenericFactory<string> GenFact = new GenericFactory<string>();
    // Fill the list of objects.
    for (Loop = 0; Loop < Iters; Loop++)
    {
       RC = new RedColor(Loop);
       GenList.Add(RC);
    }
    // Clean up before starting.
    GC.Collect();
    // Begin the for next loop test.
    var Watch = new Stopwatch();
    Watch.Start();
    for (int i = 0; i < Iters; i++)
    {
       RC = GenList[i];
       RC.TestValue = RC.TestValue + 1;
    }
    Watch.Stop();
    Console.WriteLine("Time Elapsed for the for loop = {0} ms", Watch.ElapsedMilliseconds);
    // Begin the foreach loop test.
    Watch = new Stopwatch();
    Watch.Start();
    foreach(RedColor RC1 in GenList)
    {
        RC1.TestValue = RC1.TestValue + 1;
    }
    Watch.Stop();
    Console.WriteLine("Time Elapsed for the for next loop = {0} ms - press a key to continue.", Watch.ElapsedMilliseconds);
    Console.ReadKey();
}

Bitmap Sort – Most Efficient Method To Sort Integers

A bitmap sort is the fastest and most efficient way to sort a limited set of integers.  It was first published by Jon Bentley in the book Programming Pearls.  It works by thinking of a chunk of memory as a set of numbered bits.  Each number to be sorted should be thought of as a bit in that chunk of memory.  So, after all the bits are set, then it is a simple matter to obtain the sorted numbers by starting from the beginning (or the end if reverse order is required) and checking if a bit is set.  If so, then it is moved to the output buffer.  The following code has been tested and performs this task:

public static void BitMapSort(int[] InBuf, int[] OutBuf, int MaxInt)
{
   // This function demonstrates how to efficiently sort an array of numbers.
   // Assumes that each element is >= 0.
   // InBuf holds the numbers to be sorted.
   // OutBuf returns the numbers in sorted order.
   // MaxInt identifies the largest number that can be encountered, which is a limitation
   // to this sort technique.
   //
   int Loop, Val, OutIndex;
   //
   int InBufSize = InBuf.Length;
   // Init the bit array to hold enough bits based upon the biggest int.  Initizlied by default to false.
   BitArray BA = new BitArray(MaxInt + 1);
   // Set the corresponding bit in the bit array to the number in InBuf.
   for (Loop = 0; Loop < InBufSize; Loop++)
   {
      Val = InBuf[Loop];
      BA.Set(Val, true);
   }
   // Get each bit set and move it into OutBuf in sorted order.
   OutIndex = 0;
   for (Loop = 0; Loop <= MaxInt; Loop++)
   {
      if (BA.Get(Loop))
      {
         OutBuf[OutIndex++] = Loop;
      }
   }
}

 

The code to test out this function looks like this:

private void TestBitmapSort()
{
   // This function tests out the bitmap sort function.
   Random RandObj = new Random(12345);
   int[] InBuf = new int[1000];
   int[] OutBuf = new int[1000];
   int MaxInt = 75000;
   int Loop, Val;
   // Init the InBuf with random numbers between 0 and
   BitArray BA = new BitArray(MaxInt + 1);
   for (Loop = 0; Loop < 1000; Loop++)
   {
      // Avoid duplicate numbers.
      for (;;)
      {
        Val = RandObj.Next(MaxInt + 1);
        if (BA.Get(Val))
          continue;
        InBuf[Loop] = Val;
        BA.Set(Val, true);
        break;
      }
   }
   FastEvaluate.BitMapSort(InBuf, OutBuf, MaxInt);
   for (Loop = 0; Loop < 999; Loop++)
   {
      if (OutBuf[Loop] >= OutBuf[Loop+1])
      {
        LogMessage("TestBitmapSort failed");
        break;
      }
   }
}

Efficient File I/O From C#

This article describes and benchmarks different ways to do file I/O from C#.  All of the code referenced in this article is available for download and is free to use.

There are many different ways to do file I/O in C#.  The .NET framework provides the following classes:

• File – Provides static methods for the creation, opening of files, copying, deleting, moving, reading, and writing of files.
• BinaryReader – Reads primitive data types as binary values using a specific encoding.
• BinaryWriter – Writes primitive types in binary to a stream and supports writing strings using a specific encoding.
• StreamReader – Implements a TextReader that reads characters from a byte stream using a particular encoding.
• StreamWriter – Implements a TextWriter for writing characters to a stream using a particular encoding.
• FileStream   – Exposes a Stream around a file, supporting both synchronous and asynchronous read and write operations.

The Windows operating system also provides at least two functions to read and write files:

• ReadFile  – Reads data from a file.  This function supports both synchronous and asynchronous operations.
• WriteFile – Writes data to a file. This function supports both synchronous and asynchronous operation.

One small problem with using the Windows ReadFile and WriteFile functions is that the interface requires using pointers.  In C#, pointers can only be used in a method or class declared unsafe.  So, since some organizations are somewhat leery of using pointers and unsafe code, all of this code has been put into its own class and the calling methods do not have to declared unsafe.  The name of this class is named:

WinFileIO – provides the capability to utilize the ReadFile and Writefile windows IO functions.  Based upon extensive testing, these functions are the most efficient way to perform file I/O from C# and C++.

How The Test Program Works

The user interface provides the following buttons:

• Run All – tests the Read File and Write File functions.  Similar to pressing the Read File and Write File buttons.
• Read File – tests the read file methods for each class listed above.  Each test consists of reading in 3 text files with sizes of roughly < 1mb, 10mb, and 50mb.
• Write File – tests the write file methods for each class listed above.  Each test consists of reading in 3 text files with sizes of roughly < 1mb, 10mb, and 50mb.
• WinFileIO Unit Tests – tests each public interface I/O method in the class to show that it works correctly.

Testing Methodology:

Each test is consistent in the way benchmarking is done.  For each test function, the current time is obtained before the I/O function begins, and is then retrieved right after the test ends.  Benchmarking includes the time it takes to open and close the file, except for the last set of tests which only measure the time it takes to read or write the files.

Each test consists of reading in or writing out 3 files:

• < 1 MB – approximately .66 megabytes.
• 10 MB – approximately 10 megabytes.
• 50 MB – approximately 50 megabytes.

The test machine consists of the following parts:

• CPU – INTEL i7-950.
• Memory – 6 GB
• OS – Windows 7 Pro which is contained on a solid state drive.
• Hard drive –  Western Digital 1TB SATA III 6GB/S 7200RPM 64MB CACHE, which is where the data files reside.
• IDE – Visual Studio 2008 standard edition with .NET framework version 3.5 SP1.

Read Methods Tested:

File class methods:

• ReadAllLines – reads all lines of the file into a string array.  See TestReadAllLines.
• ReadAllText – reads the entire contents of the file into a string.  See TestReadAllText.
• ReadAllBytes – reads the entire contents of the file into a byte array. See TestReadAllBytes.

BinaryReader methods:

• Read – reads the entire contents of the file into a character array.  See TestBinaryReader1.

StreamReader methods:

Read(1) – reads the entire contents of the file into a character array using the single argument constructor.  See TestStreamReader1.

• Read(2) – reads the entire contents of the file into a character array.  Uses a constructor to specify a sector aligned buffer size.  See TestStreamReader2.
• Read(3) – a loop is used to read in the entire contents of the file into a character array.  Uses a constructor to specify a sector aligned buffer size.  The loop is terminated when the Peek function indicates there is no more data.
• ReadBlock – reads the entire contents of the file into a character array.  Uses a constructor to specify a sector aligned buffer size.  See TestStreamReader5.
• ReadToEnd – a loop is used to read in the entire contents of the file into a character array.  Uses a constructor to specify a sector aligned buffer size.  The loop is terminated when the Peek function indicates there is no more data.  See TestStreamReader4.
• ReadBlock – reads the entire contents of the file into a character array.  Uses a constructor to specify a sector aligned buffer   See TestStreamReader3.
• ReadLine – reads the entire contents of the file line by line into a string.  Uses a constructor to specify a sector aligned buffer size.  See TestStreamReader3.

FileStream methods:

• Read(1) – reads the entire contents of the file into a byte array.  See TestFileStreamRead1.
• Read(2) – reads the entire contents of the file into a byte array and parse into lines.  See TestFileStreamRead2.
• Read(3) – reads the entire contents of the file into a byte array and parse into lines.  See TestFileStreamRead2A.
• Read(4) – reads the entire contents of the file into a byte array.  Uses the RandomAccess option in the constructor to determine if there is any impact on performance.  See TestFileStreamRead3.
• BeginRead(1) – reads the entire contents of the file into a byte array asynchronously.  See TestFileStreamRead4.
• BeginRead(2) – reads the entire contents of the file into a byte array asynchronously and parses into lines.  See TestFileStreamRead5.
• BeginRead(3) – reads the entire contents of the file into a byte array asynchronously and parses into lines.  Identical to TestFileStreamRead5 except for using a different threading locking mechanism.  See TestFileStreamRead6.
• BeginRead(4) – reads the entire contents of the file into a byte array asynchronously.  Uses no locking mechanism.  See TestFileStreamRead7.
• BeginRead(5) – reads the entire contents of the file into a byte array asynchronously and parses into lines.  See TestFileStreamRead8.

WinFileIO methods:

• Read – reads the specified number of bytes into an array.
• ReadUntilEOF – reads the entire contents of a file into an array.
• ReadBlocks – reads the specified number of bytes into an array.

Write Methods Tested:

File class methods:

• WriteAllLines – writes all lines in a string array to a file.  See TestWriteAllLines.
• WriteAllText – writes the entire contents in a string to a file.  See TestWriteAllText.
• WriteAllBytes – writes the entire contents of a byte buffer to a file.  See TestWriteAllBytes.

BinaryWriter methods:

• Write – writes the entire contents of a character array to a file.  See TestBinaryWriter1.

StreamWriter methods:

• Write – writes the entire contents a character array to a file.  See TestStreamWriter1.

FileStream methods:

• Write(1) – writes the entire contents of a byte array to a file.  See TestFileStreamWrite1.
• Write(2) – writes the entire contents of a byte array to a file.  See TestFileStreamWrite2.
• Write(3) – writes the entire contents of a byte array to a file.  See TestFileStreamWrite3.

WinFileIO methods:

• Write – writes the entire contents of an array to a file.  See TestWriteFileWinAPI1.
• WriteBlocks – writes the entire contents of an array to a file.  See TestWriteFileWinAPI2.

WinFileIO Class:

This class was designed to make it easy to use the Windows ReadFile and WriteFile methods.  It handles all of the unsafe operations with pointers.  Calling methods do not have to be declared unsafe.  Implements the IDisposable interface which means that the Dispose method should be called when the object is no longer needed.  If there is a problem with any method in this class, it will throw an exception with the Windows error information.  If the function returns, then this indicates success.  The only exception to this is the Close method.

Constructors:

• WinFileIO() – default.  If this constructor is used, then the PinBuffer function must be called.
• WinFileIO(Array Buffer) – this constructor should be used most of the time.  The Buffer is used to read in or write out the data.  The array passed in can be of any type provided it does not contain any references or pointers.  So, byte, char, int, long, and double arrays should all work.  But string arrays will not since strings use pointers.  The code has only been tested with byte arrays.

Methods:

• void PinBuffer(Array Buffer) – this method pins the buffer in memory and retrieves a pointer to it which is used for all I/O operations.  UnpinBuffer is called by this function so it need not be called by the user.  This function only needs to be called if the default constructor is used or a different buffer needs to be used for reading or writing.
• void OpenForReading(string FileName) – opens a file for reading.  The argument FileName must contain the path and filename of the file to be read.
• void OpenForWriting(string FileName) – opens a file for writing.  If the file exists, it will be overwritten.
• int Read(int BytesToRead) – reads in a file up to BytesToRead The return value is the number of bytes read.  BytesToRead must not be larger than the size of the buffer specified in the constructor or PinBuffer.
• int ReadUntilEOF() – reads in the entire contents of the file.  The file must be <= 2GB.  If the buffer is not large enough to read the file, then an ApplicationException will be thrown.  No check is made to see if the buffer is large enough to hold the file.  If this is needed, then use the ReadBlocks method.
• int ReadBlocks(int BytesToRead) – reads a total of BytesToRead at a time.  There is a limit of 2gb per call.  BytesToRead should not be larger than the size of the buffer specified in the constructor or PinBuffer.
• int Write(int BytesToWrite) – writes a buffer out to a file.  The return value is the number of bytes written to the file.
• int WriteBlocks(int NumBytesToWrite) – writes a buffer out to a file.  The return value is the number of bytes written to the file.
• bool Close() – closes the file.  If this method succeeds, then true is returned.  Otherwise, false is returned.

BenchMark Read Results::

Running the read file tests:
Total time reading < 1MB with File.ReadAllLines                            = 00:00:00.0030002
Total time reading 10MB with File.ReadAllLines                             = 00:00:00.0640037
Total time reading 50MB with File.ReadAllLines                             = 00:00:00.3540202
Total time reading < 1MB with File.ReadAllText                             = 00:00:00.0040002
Total time reading 10MB with File.ReadAllText                              = 00:00:00.0360020
Total time reading 50MB with File.ReadAllText                              = 00:00:00.1630093
Total time reading < 1MB with File.ReadAllBytes                            = 00:00:00
Total time reading 10MB with File.ReadAllBytes                             = 00:00:00.0050003
Total time reading 50MB with File.ReadAllBytes                             = 00:00:00.0260015

Total time reading < 1MB with BinaryReader.Read                         = 00:00:00.0020001
Total time reading 10MB with BinaryReader.Read                          = 00:00:00.0270016
Total time reading 50MB with BinaryReader.Read                          = 00:00:00.1260072

Total time reading < 1MB with StreamReader1.Read                      = 00:00:00.0010001
Total time reading 10MB with StreamReader1.Read                       = 00:00:00.0200011
Total time reading 50MB with StreamReader1.Read                       = 00:00:00.0960055
Total time reading < 1MB with StreamReader2.Read(large buf)    = 00:00:00.0010001
Total time reading 10MB with StreamReader2.Read(large buf)     = 00:00:00.0160009
Total time reading 50MB with StreamReader2.Read(large buf)     = 00:00:00.0750043
Total time reading < 1MB with StreamReader3.ReadBlock             = 00:00:00.0010001
Total time reading 10MB with StreamReader3.ReadBlock              = 00:00:00.0150008
Total time reading 50MB with StreamReader3.ReadBlock              = 00:00:00.0750043
Total time reading < 1MB with StreamReader4.ReadToEnd           = 00:00:00.0020001
Total time reading 10MB with StreamReader4.ReadToEnd            = 00:00:00.0320018
Total time reading 50MB with StreamReader4.ReadToEnd            = 00:00:00.1720099
Total time reading < 1MB with mult StreamReader5.Read              = 00:00:00.0020001
Total time reading 10MB with mult StreamReader5.Read               = 00:00:00.0430025
Total time reading 50MB with mult StreamReader5.Read               = 00:00:00.0850048
Total time reading < 1MB with StreamReader6.ReadLine                = 00:00:00.0020002
Total time reading 10MB with StreamReader6.ReadLine                 = 00:00:00.0310017
Total time reading 50MB with StreamReader6.ReadLine                 = 00:00:00.1510087
Total time reading < 1MB with StreamReader7.Read parsing          = 00:00:00.1470084
Total time reading 10MB with StreamReader7.Read parsing           = 00:00:00.1600091
Total time reading 50MB with StreamReader7.Read parsing           = 00:00:00.2260129

Total time reading < 1MB with FileStream1.Read no parsing           = 00:00:00.0080005
Total time reading 10MB with FileStream1.Read no parsing            = 00:00:00.0040002
Total time reading 50MB with FileStream1.Read no parsing            = 00:00:00.0190011
Total time reading < 1MB with FileStream2.Read parsing                 = 00:00:00.1220070
Total time reading 10MB with FileStream2.Read parsing                  = 00:00:00.1220069
Total time reading 50MB with FileStream2.Read parsing                  = 00:00:00.1370079
Total time reading < 1MB with multiFileStream2A.Read parsing     = 00:00:00.1180067
Total time reading 10MB with multiFileStream2A.Read parsing      = 00:00:00.1210070
Total time reading 50MB with multiFileStream2A.Read parsing      = 00:00:00.1320075
Total time reading < 1MB with FileStream3.Read(Rand) no parsing= 00:00:00
Total time reading 10MB with FileStream3.Read(Rand) no parsing = 0:00:00.0030002
Total time reading 50MB with FileStream3.Read(Rand) no parsing = 00:00:00.0170009
Total time reading < 1MB with FileStream4.BeginRead no parsing  = 0:00:00.0020001
Total time reading 10MB with FileStream4.BeginRead no parsing   = 0:00:00.0040002
Total time reading 50MB with FileStream4.BeginRead no parsing   = 00:00:00.0180011
Total time reading < 1MB with FileStream5.BeginRead parsing        = 0:00:00.0020001
Total time reading 10MB with FileStream5.BeginRead parsing         = 0:00:00.0280016
Total time reading 50MB with FileStream5.BeginRead parsing         = 0:00:00.1370079
Total time reading < 1MB with FileStream6.BeginRead parsing       = 00:00:00.0030002
Total time reading 10MB with FileStream6.BeginRead parsing         = 00:00:00.0280016
Total time reading 50MB with FileStream6.BeginRead parsing         = 00:00:00.1360077
Total time reading < 1MB with FileStream7.BeginRead                       = 00:00:00
Total time reading 10MB with FileStream7.BeginRead                      = 00:00:00.0050003
Total time reading 50MB with FileStream7.BeginRead                      = 00:00:00.0240014
Total time reading < 1MB with FileStream8.BeginRead parsing       = 00:00:00.0020001
Total time reading 10MB with FileStream8.BeginRead parsing        = 00:00:00.0310018
Total time reading 50MB with FileStream8.BeginRead parsing        = 00:00:00.1480085

Total time reading < 1MB with WFIO1.Read No Parsing                   = 00:00:00.0020001
Total time reading 10MB with WFIO1.Read No Parsing                    = 00:00:00.0020001
Total time reading 50MB with WFIO1.Read No Parsing                    = 00:00:00.0120007
Total time reading < 1MB with WFIO2.ReadUntilEOF No Parsing  = 00:00:00.0010001
Total time reading 10MB with WFIO2.ReadUntilEOF No Parsing   = 00:00:00.0030001
Total time reading 50MB with WFIO2.ReadUntilEOF No Parsing   = 00:00:00.0140008
Total time reading < 1MB with WFIO3.ReadBlocks API No Parsing= 00:00:00.0010001
Total time reading 10MB with WFIO3.ReadBlocks API No Parsing = 00:00:00.0030002
Total time reading 50MB with WFIO3.ReadBlocks API No Parsing = 00:00:00.0130008

Total time reading < 1MB with BinaryReader.Read                            = 00:00:00.0010001
Total time reading 10MB with BinaryReader.Read                             = 00:00:00.0220012
Total time reading 50MB with BinaryReader.Read                             = 00:00:00.1080062
Total time reading < 1MB with StreamReader2.Read(large buf)      = 00:00:00.0010001
Total time reading 10MB with StreamReader2.Read(large buf)       = 00:00:00.0150008
Total time reading 50MB with StreamReader2.Read(large buf)       = 00:00:00.0690040
Total time reading < 1MB with FileStream1.Read no parsing            = 00:00:00.0010000
Total time reading 10MB with FileStream1.Read no parsing             = 00:00:00.0030002
Total time reading 50MB with FileStream1.Read no parsing             = 00:00:00.0130008
Total time reading < 1MB with WFIO.Read No Open/Close              = 00:00:00.0010001
Total time reading 10MB with WFIO.Read No Open/Close               = 00:00:00.0030001
Total time reading 50MB with WFIO.Read No Open/Close               = 00:00:00.0130008
Read file tests have completed.

Analysis Of Read Results:

The File class provides the simplest way to read in a class.  The ReadAllBytes method of this class provides a fairly efficient way to read in a file and is only bested by the read methods in the FileStream and WinFileIO classes.  From the results, it appears that the best StreamReader and BinaryReader read methods are roughly 3 to 5 times slower than the ReadAllBytes method.

The FileStream read methods were shown to be the fastest way to read a file into memory using a method from the .NET Framework.  The synchronous method of reading the entire file into memory in TestFileStreamRead1 and TestFileStreamRead3 proved to be the best of this set of tests with TestFileStreamRead3 taking top honors by a hair.  The only difference between these two tests is that the file is opened with the SequentialScan option in TestFileStreamRead1 .vs. opening the file with RandomAccess in TestFileStreamRead3.  Since there are always other OS activities going on while running a benchmark, it is hard to know if one method is superior to another when it is this close.  However, these tests have been tested on other systems multiple times with different Windows OSs with the same results, so in this case it appears that the TestFileStreamRead3 method is marginally superior.

The biggest disappointment came with the 5 FileStream asynchronous tests given in TestFileStreamRead4 – TestFileStreamRead8.  These tests all show that reading in a file asynchronously is inferior to reading it in synchronously.  This is even true if other activities like parsing a file is done in between reads.  For example, compare the results of TestFileStreamRead2A which reads in a file synchronously and parses the data against the results of TestFileStreamRead5 which reads in a file asynchronously and parses the data while the next block is read in asynchronously.  Even when the locks have been removed (see TestFileStreamRead8), it is still at least 10% slower than reading the file in synchronously and then parsing the file afterwards (see TestFileStreamRead2A).

The WinFileIO class proved to be the fastest way to read a file in.  It is between 33% to 50% faster than the fastest FileStream read method based upon the measured times above.  However, the last set of tests – TestBinaryReader1NoOpenClose through TestReadFileWinAPINoOpenClose measure how quickly the files are read in after the file is opened.  According to the results, the FileStream read method is just as fast as any of the WinFileIO read methods.  So, it looks like the .NET framework takes longer to open a file than the windows CreateFile function.

Benchmark Write Results:

Running the write file Tests:
Total time writing < 1MB with File.WriteAllLines                             = 00:00:00.0050003
Total time writing 10MB with File.WriteAllLines                              = 00:00:00.0350020
Total time writing 50MB with File.WriteAllLines                              = 00:00:00.1620093
Total time writing < 1MB with File.TestWriteAllText                      = 00:00:00.0040002
Total time writing 10MB with File.TestWriteAllText                       = 00:00:00.0270016
Total time writing 50MB with File.TestWriteAllText                       = 00:00:00.1440082
Total time writing < 1MB with File.WriteAllBytes                             = 00:00:00.3560204
Total time writing 10MB with File.WriteAllBytes                              = 00:00:00.3390194
Total time writing 50MB with File.WriteAllBytes                              = 00:00:00.3530202

Total time writing < 1MB with BinaryWriter.Write                           = 00:00:00.0010001
Total time writing 10MB with BinaryWriter.Write                            = 00:00:00.0050003
Total time writing 50MB with BinaryWriter.Write                            = 00:00:00.3040174

Total time writing < 1MB with StreamWriter1.Write                        = 00:00:00.0030002
Total time writing 10MB with StreamWriter1.Write                         = 00:00:00.0230013
Total time writing 50MB with StreamWriter1.Write                         = 00:00:00.1140065

Total time writing < 1MB with FileStream1.Write no parsing          = 00:00:00.0010001
Total time writing 10MB with FileStream1.Write no parsing           = 00:00:00.0050003
Total time writing 50MB with FileStream1.Write no parsing           = 00:00:00.3670210
Total time writing < 1MB with FileStream2.Write no parsing          = 00:00:00.0070004
Total time writing 10MB with FileStream2.Write no parsing           = 00:00:00.1060061
Total time writing 50MB with FileStream2.Write no parsing           = 00:00:00.5000286
Total time writing < 1MB with FileStream3.Write no parsing          = 00:00:00.0100006
Total time writing 10MB with FileStream3.Write no parsing           = 00:00:00.1150066
Total time writing 50MB with FileStream3.Write no parsing           = 00:00:00.5840334

Total time writing < 1MB with WFIO1.Write No Parsing                  = 00:00:00.0020001
Total time writing 10MB with WFIO1.Write No Parsing                   = 00:00:00.0050003
Total time writing 50MB with WFIO1.Write No Parsing                   = 00:00:00.3530202
Total time writing < 1MB with WFIO2.WriteBlocks No Parsing       = 00:00:00.0010001
Total time writing 10MB with WFIO2.WriteBlocks No Parsing        = 00:00:00.0060003
Total time writing 50MB with WFIO2.WriteBlocks No Parsing        = 00:00:00.0260015
Write file tests have completed.

ANALYSIS OF WRITE RESULTS:

The File class provides the simplest way to write a file out.  Unlike the ReadAllBytes method for reading files, WriteAllBytes is less efficient than WriteAllText according to the results above.

One interesting result is that the times to write out the < 1MB file and 10 MB file for the BinaryWriter, FileStream, and WinFileIO classes are quite similar and fast.  However, the time to write out the 50 MB file takes around 60 times longer than the 10 MB file.  This does not apply to the WinFileIO.WriteBlocks method which proved to be the fastest way to write a file out.  The most likely reason for this is that WriteBlocks writes the file out in 65,536 byte chunks.  However, the TestFileStreamWrite3 test also writes out the file in 65,536 byte chunks and proved to be the slowest method.  I can’t think of a good explanation for this other than perhaps the FileStream.Write method has some issues.

Conclusion:

Any time a benchmark is done trying to test out file I/O methods, it is very difficult to completely trust the results due to the operating system caching files and other OS activities.  If different files are used, then they can be placed on areas of the drive that will yield better performance simply because the drive can access them faster and can impact the results.  So, take a little grain of salt with these benchmark results.  To achieve the best performance for your environment, I would recommend trying out different classes in your production environment to see which yields the best performance.

Having said that and after testing on 3 different machines with similar results, I believe that the best performing file I/O can be obtained from the WinFileIO read methods for reading a file and the WinFileIO.WriteBlocks method for writing files.

I have done similar tests with C++ which are not shown here and believe that the Windows ReadFile and WriteFile methods are the most efficient way to do file I/O from that language as well.

Download and installation:

Click this link.  This file is a zipped file containing 2 visual studio projects.  Extract it into the folder of your choice and leave the folder hierarchy intact.  The code was built and tested with Visual Studio 2008, but it should work with Visual Studio 2015 with little if any modification.  To make it work with previous versions of Visual Studio, you may have to open a new project and  add the individual files to each project.

The following code files are contained in the in the FileTestsForEfficiency folder:

• TestsForEfficiency.cs – entry point to the application.
• MainForm.cs – holds the UI designer and button events.
• FileEffTests.cs – holds the file I/O benchmark tests, which is contained in the FileEfficientTests class.
• Win32FileIO.cs – holds the class used to implement the Windows ReadFile and WriteFile functionality.
• WinFileIOUnitTests.cs – holds the unit tests that test out the I/O methods of the WinFileIO class.

The following code files are contained in the FileTestsForEfficiency folder:

• Win32FileIO.cs – holds the class used to implement file I/O using the Windows ReadFile and Writefile methods.

Introduction.

Hello!  Thank you for visiting my blog.  My name is Bob Bryan and I have been developing software for over 20 years.

This blog is about efficient software design, implementation, and testing.  This includes bench marking software as well as interesting code examples.

Please feel free to visit my other blog:

Tablizing the binomial coefficient – which explains how a model constrained by the binomial coefficient can be placed into a table and how that table can be accessed easily, efficiently, and without using large amounts of memory. A download file is also provided that implements the solution as a C# generic class as well as a test class that shows how it can be used and proves that it works as intended.

I can be reached by email at: RobertGBryanREMOVETHIS@yahoo.com.  In order to avoid automated spam messages, the above email address is spoofed.  Please remove REMOVETHIS from the email address.  For example, if my email address was abcREMOVETHIS@yahoo.com, then the actual email address would be abc@yahoo.com.

 

 Thanks for stopping by.