Latest Event Updates

Run time initialisation of protobuf-net v2 without annotation

Posted on

I have been using protobuf-net for the project at office but it’s only implemented for a limited number of entities (or business objects). We’re now trying to see how to implement this to the whole entities that we’re sending back n forth between the UI and the back end.

As we’re currently using a code generation “tool” to produce the code, we’re thinking to incorporate the annotation/attribute into the generated class/property code, however due to some not really technical reason, this is a bit complicated to achieve.

I thought of checking the site again to get some idea and I came across to this statement “allow use without attributes if you wish” on the page.

Then I tried to dig out more on the SO and found some more good leads on this. Apparently it’s very much possible to perform “registration” during run time mode which allows us NOT to use any annotation on the entities.

So thanks to the pointers, there are 2 alternatives for this:
1. With very minimal annotation on the class level

[ProtoContract(ImplicitFields = ImplicitFields.AllPublic)] // only required on the class level
class PersonEntity
{
   public string FirstName { get; set; }
   public string LastName { get; set; }
}

2. Without any annotation (using RuntimeTypeModel)

static void InitializeProtobufRunTime()
{
    var assembly = Assembly.GetAssembly(typeof(PlainEntities.PersonEntity));
    var types = assembly.GetTypes();
    foreach (var t in types.Where(x => x.Namespace.Contains("PlainEntities")))
    {
        Console.WriteLine("Processing {0}", t.FullName);
        var meta = RuntimeTypeModel.Default.Add(t, false);
        var index = 1;

        // find any derived class for the entity
        foreach (var d in types.Where(x => x.IsSubclassOf(t)))
        {
            var i = index++;
            Console.WriteLine("\tSubtype: {0} - #{1}", d.Name, i);
            meta.AddSubType(i, d);
        }

        // then add the properties
        foreach (var p in t.GetProperties(BindingFlags.Instance | BindingFlags.Public | BindingFlags.DeclaredOnly).Where(x => x.GetSetMethod() != null))
        {
            var i = index++;
            Console.WriteLine("\tProperty: {0} - #{1}", p.Name, i);
            meta.AddField(i, p.Name);
        }
    }
}

And both the above works quite well without any performance differences.


------------------
TestBinaryEntities
------------------
Process: 100000 items, MemorySize: 7400705, Completed in: 3877 ms, Serialization took: 676 ms, Deserialization took: 2948 ms

----------------------------------
TestProtobufFullyAnnotatedEntities
----------------------------------
Process: 100000 items, MemorySize: 3983490, Completed in: 682 ms, Serialization took: 164 ms, Deserialization took: 253 ms

-------------------------------------
TestProtobufImplicitAnnotatedEntities
-------------------------------------
Process: 100000 items, MemorySize: 3983490, Completed in: 595 ms, Serialization took: 104 ms, Deserialization took: 210 ms

-------------------------------
TestProtobufRuntimeRegistration
-------------------------------
Processing ProtobufTestConsole.PlainEntities.BaseEntity
Subtype: PersonEntity - #1
Property: Id - #2
Property: Gender - #3
Processing ProtobufTestConsole.PlainEntities.PersonEntity
Property: FirstName - #1
Property: LastName - #2
Property: Age - #3
Process: 100000 items, MemorySize: 4083490, Completed in: 646 ms, Serialization took: 113 ms, Deserialization took: 232 ms

Looking forward to get this in :)

Also attached the sample project for reference

Sample Project

Why Task Parallel Library (TPL) = Great stuffs

Posted on

I just came back to office today from a short staycation and while going through the email back logs, I saw an email from a colleague asking for my practical experience with the Task Parallel Library (TPL) so far.

I’m quite delighted to share some with him and thought of just dropping some to here as well ;)

Practical usages:

  • Parallel data retrieval from different sources
    This has saved significant amount of time, e.g. we have 3 data retrievals, the 1st takes 10 seconds, the 2nd takes 7 seconds and the 3rd takes 4 seconds, rather than 21 seconds sequentially, we can get all 3 results in 10 (max) now.
  • Parallel data submission with limited concurrency
    With long running processes, we need to be careful not to overload the back end services, so when TPL comes with Limited Concurrency Task Scheduler, this becomes a life saver and we can remove all the previous code complexities to achieve the same.

Summaries:

  • The Task has greatly simplified the developer’s code to be able to perform asynchronous operation which returns values where previously it’s not straight forward to achieve the same (e.g. creaging a wrapper using threadpool)
  • Out of the box / It comes with the .Net 4 (directly shipped by microsoft), this very much makes TPL available for us (developers) and the users (client) without worrying a separate component to download
  • More good stuffs are on the way with latest .Net 4.5 Framework

The downsides:

  • Potential parallelism -> good stuff + not too good stuff, good stuff -> we don’t need to worry, not too good stuff -> there are times that we want it to happen in parallel but it’s too clever not to do parallel in heavy usage especially in low spec machine (e.g. client machine is not as good as our developer PC)
  • Unit testing -> It’s not so easy to test our code if we start mixing TPL into our code, although it’s true that we should separate the logic bit from the concurrency, but in reality sometimes we end up with unnecessary code to make it testable. I do hope more improvements to come for this, there are many being discussed in stackoverflow

Generate Func by using Expression Trees

Posted on Updated on

I stumbled in a code change today which involved some value look-up in a dictionary based on some property name which is by convention (paramA + paramB + “property”).

So there was used to be a pre-calculated sum stored in the dictionary by a user control.

class SomeClass
{
    public Decimal DailyFunctionalAmount { get; set; }
    public Decimal MonthToDateFunctionalAmount { get; set; }
    ///..... have much more here
}

Decimal GetTotal(String fieldName, IDictionary<String, Decimal> lookups)
{
    // used to be as simple as getting the pre-calculated sum via dictionary (populated somewhere else by control)
    var totalByLookup = lookups[fieldName];
    return totalByLookup;
}

Now, I need to rely on the underlying data rows to calculate the sum.

var lists = new List<SomeClass> {
    new SomeClass { DailyFunctionalAmount = 10, MonthToDateFunctionalAmount = 100},
    new SomeClass { DailyFunctionalAmount = 10, MonthToDateFunctionalAmount = 100},
};

// I could not use the following code as this is compiled/pre-determined code where we actually need the selector in run-time based on the generated property name by convention
var totalByLinq = lists.Sum(r => r.DailyFunctionalAmount); 

Luckily, there is this expression trees which seems to be perfect for the job, rather than a nasty static function returning selector based on possible combinations for the property name by convention.

// returns -> dtoType => dtoType.propertyName
Func<SomeClass, Decimal> GetMyFunc(string propertyName)
{
    var dtoTypeParameter = Expression.Parameter(typeof(SomeClass), "dtoType");
    var dtoPropertySelector = Expression.Property(dtoTypeParameter, propertyName);
    var lamdaExpression = Expression.Lambda<Func<SomeClass, Decimal>>(dtoPropertySelector, new ParameterExpression[] { dtoTypeParameter });

    return lamdaExpression.Compile();
}

Decimal GetTotal(String fieldName, IEnumerable<SomeClass> lists)
{
    // solution - generate the func in run-time
    var runTimeSelector = GetMyFunc(fieldName);
    var totalByLinq = lists.Sum(runTimeSelector);
    return totalByLinq;
}

It’s surely world of possibilities with this expression trees ;)

Full source code below:

[TestClass]
public class UnitTest1
{
    [TestMethod]
    public void TestMethod1()
    {
        var lists = new List<SomeClass> {
            new SomeClass { DailyFunctionalAmount = 10, MonthToDateFunctionalAmount = 100},
            new SomeClass { DailyFunctionalAmount = 10, MonthToDateFunctionalAmount = 100},
        };

        var lookups = new Dictionary<String, Decimal>(); // stored with the property name as key in the dictionary and the sum as value
        lookups.Add("DailyFunctionalAmount", 20);
        lookups.Add("MonthToDateFunctionalAmount", 200);

        var dailyFunctionalFieldName = GetFieldName("Daily", "Functional");
        var monthToDateFunctionalFieldName = GetFieldName("MonthToDate", "Functional");

        // old way
        Assert.AreEqual(20, GetTotal(dailyFunctionalFieldName, lookups));
        Assert.AreEqual(200, GetTotal(monthToDateFunctionalFieldName, lookups));

        // new way 
        Assert.AreEqual(20, GetTotal(dailyFunctionalFieldName, lists));
        Assert.AreEqual(200, GetTotal(monthToDateFunctionalFieldName, lists));
    }

    Decimal GetTotal(String fieldName, IDictionary<String, Decimal> lookups)
    {
        // used to be as simple as getting the pre-calculated sum via dictionary (populated somewhere else by control)
        var totalByLookup = lookups[fieldName];
        return totalByLookup;
    }

    Decimal GetTotal(String fieldName, IEnumerable<SomeClass> lists)
    {
        // now it needs to be manually calculated from the list

        //var totalByLinq = lists.Sum(r => r.DailyFunctionalAmount); // but this is compiled/pre-determined code where we actually need the selector in run-time

        // solution - generate the func in run-time
        var runTimeSelector = GetMyFunc(fieldName);
        var totalByLinq = lists.Sum(runTimeSelector);
        return totalByLinq;
    }

    String GetFieldName(String periodic, String currency)
    {
        // the field name is generated by some convention [Daily/MonthToDate] + [Functional/Transaction] + "Amount"
        return periodic + currency + "Amount";
    }

    // returns -> dtoType => dtoType.propertyName
    Func<SomeClass, Decimal> GetMyFunc(string propertyName)
    {
        var dtoTypeParameter = Expression.Parameter(typeof(SomeClass), "dtoType");
        var dtoPropertySelector = Expression.Property(dtoTypeParameter, propertyName);
        var lamdaExpression = Expression.Lambda<Func<SomeClass, Decimal>>(dtoPropertySelector, new ParameterExpression[] { dtoTypeParameter });

        return lamdaExpression.Compile();
    }
}

class SomeClass
{
    public Decimal DailyFunctionalAmount { get; set; }
    public Decimal MonthToDateFunctionalAmount { get; set; }
    ///..... have much more here
}

Blending with Sample Data feature (Blend 4)

Posted on

Had a chance to play more with Blend 4 this week, have been eyeing on the Sample Data feature for a while but only had a chance to use it in practice now.

So imagine if you have defined the backing view model below

public class MainViewModel
{
	public MainViewModel()
	{
		// Insert code required on object creation below this point.
	}
	public IList<Person> Persons { get; set; }
}

public abstract class Person
{
	public String Name { get; set; }
	public Int32 Age { get; set; }
	public String Initials { get{ return "some initials"; } }
}

public class Male : Person
{
}

public class Female : Person
{
}

You can then open the view, on the top right panel, click on the Create Sample Data from Class option below

Choose the view model to generate the sample data, click Ok

Bam! the sample data file has been generated nicely below, and notice one thing, the read only Initials property can be “set” as well there :)


Before moving on to the next step, be sure to replace Person Type on the rows into either Male or Female.

Now, time to get into the view with the d:DataContext binding and data templates by Data Type for each Male and Female.

hhhmm… a little bit disappointing, isn’t it? all are seem to be in place, but why it’s displaying these weirds _.di12.NameSpace.Class rather than our expected data template result?
Now back to the xaml, add d:IsDesignTimeCreatable=”False” attribute on both data template elements, save, close and reopen the xaml again.

And there’s the shiny UI with some sample data in it :)

Notice that Blend sometimes doesn’t refresh the design view as soon as we make some changes on the data template (it works fine on non data template changes).
I only found that the effective way is to just close and reopen to see the changes effect, so if you’re editing something on the data template and it’s not reflecting, try to close and reopen it first :)

I found this sample data feature is very useful, how bout you?

Have also shared the sample project for grabs here.

Hope this helps :)

Fallen into the pit of overconfidence

Posted on Updated on

If someone had asked me yesterday whether I’m a professional, mature, seasoned developer. I would certainly replied “Indeed, I am!”. Well, fortunately, no one did, or else I can’t imagine meeting him/her again today after the pathetic thing below.

You see, I was working on a defect/bug yesterday, it was a pretty simple bug that I was happily to fix as it looks like a less than an hour job, so I did the fix below, got the test evidences and marked that defect as fixed, ready for promotion to the test environment.

private Boolean _someFlag;
private String _someOption;

private void Execute()
{
    //... some codes here

    // the following block of codes is the fix to resolve the defect/bug
    if (_someOption.Contains("something"))
    {
        _someFlag = true;
    }
    else
    {
        _someFlag = false;
    }

    //... some codes here
}

To my horror this morning, there’s a new defect being raised, complaining that there’s an error pop-up whenever the user tried to click on the execute button.

And quickly I realized that the error was coming from the modified code above. And to make things worst, the code above is in the base class of let say 10 UI forms and 3 of them are now not working/usable at all due to this new bug.

Now fixing this new bug is another simple task to do, but the thing that stayed on my mind till now is that when I wrote that fix yesterday, it actually came to my mind that I should be putting some unit tests on them, this new bug will definitely get captured if I were to write the unit tests (basic positive, negative, null tests). But nah, my overconfidence took over that this is a simple fix, nothing’s going to ever happen with this fix, everything will be running happily tomorrow, I should check this in ASAP and hurrah! one defect down.

So what are the consequences with that?

  1. Testing activities were blocked on for 3 of 10 UIs, where actually without the code fix, all were actually working fine, it’s just a minor dodgy state that was incorrect
  2. One more day was wasted to get the new fix promoted so the testing can be continued
  3. For having end users/testers to actually see this error message, it’s embarrassing the development team
  4. And in the end, this kind of error is truly embarrassing for a .Net programmer, with these years of experiences for sure :|
The following is an excerpt taken from Uncle Bob Martin‘s The Clean Coder book.

(hope you don’t mind, Uncle Bob, I’ll make sure that I write the book review when I’m done reading your book :))

As you mature in your profession, your error rate should rapidly decrease towards the asymptote of zero. It won’t ever get to zero, but it is your responsibility to get as close as possible to it.

Hit the nail on the head of what I was expected to do. So moral of the story, be humble, don’t ignore your inner voice, and believe in unit testing/TDD.

PS: Tony, if you actually read this, just know that today in my head, i kept imagining you’re telling me “See, I told you, Bembeng” :p

New CLR Profiler for .NET Framework 4

Posted on

We just found a memory leak (again) in our latest code base that we’re going to release to production next month, seems that we’re having this issue for every release :p i think it’s unavoidable considering the amount of changes that we’re making on the app for each release ;)

Now, when I tried to do the regular steps to investigate the root cause, the CLR Profiler was not responding when I clicked the Show Heap now button and eventually display the pop-up below.

Then realised that since we have just upgraded the application to use .Net 4.0, so there’s a chance that there’s a new CLR Profiler on the block for .Net 4.0, and turned out there is actually a new one available in MSDN. And it’s not surprising, considering that .Net 4.0 comes with a new CLR as well (click on the image below to go to the msdn website for more info)

After downloaded the new CLR Profiler 4, it looks like below, notice the new Attach Process and Detach Process buttons, those will be very useful as we always need to start the application in the previous version, sweet! :)

Now, it’s time to duck into the the heap :)

Controlling the concurrency level using MaxDegreeOfParallelism in Parallel

Posted on

This is another nice available in the box feature when using the Parallel class, is the ability to basically limit the concurrency for the actions that we want to execute in parallel.

Why would we ever want to do this? wouldn’t it be best if we parallel some activities as many as possible?

For certain cases, we might want to be careful for not creating too many sudden requests at the same time, a good example will be if we’re triggering a long running process in our application server, you wouldn’t want to spawn too many requests at the same time to the server, as the server may not be able to handle so many requests at the same time and not surprisingly go down due to overload. This can be categorized as DOS attack, your back end guys will hate you for this, trust me :p

To set the concurrency level of the actions that you’re going to invoke with the Parallel class is pretty simple.

Parallel.Invoke(new ParallelOptions() { MaxDegreeOfParallelism = concurrencyLevel }, actions);

Find more about the MaxDegreeOfParallelism property in msdn.
We don’t have this such of property in the TaskCreationOptions that we can simply pass for Task, but there’s a LimitedConcurrencyLevelTaskScheduler nicely available which we can achieve the same result below where i basically spawn 10 processes with 2 max concurrency level.


15-Apr-2011 21:04:33.948 - Thead#13 - Creating 10 process definitions
15-Apr-2011 21:04:33.948 - Thead#13 - Start queueing and invoking all 10 processes
15-Apr-2011 21:04:33.948 - Thead#13 - Doing something here
15-Apr-2011 21:04:33.948 - Thead#14 - Doing something here
15-Apr-2011 21:04:34.964 - Thead#13 - Doing something here
15-Apr-2011 21:04:34.964 - Thead#14 - Doing something here
15-Apr-2011 21:04:35.964 - Thead#13 - Doing something here
15-Apr-2011 21:04:35.964 - Thead#14 - Doing something here
15-Apr-2011 21:04:36.964 - Thead#13 - Doing something here
15-Apr-2011 21:04:36.964 - Thead#14 - Doing something here
15-Apr-2011 21:04:37.964 - Thead#13 - Doing something here
15-Apr-2011 21:04:37.964 - Thead#14 - Doing something here
15-Apr-2011 21:04:38.964 - Thead#13 - All processes have been completed

using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace ParallelTests
{
    [TestClass]
    public class TaskSchedulerConcurrencyTest
    {
        [TestMethod]
        public void TestMethod1()
        {
            var p = new ProcessorUsingParallel();
            p.DoProcess(processToCreate: 10, concurrencyLevel: 2);
        }
    }

    public class ProcessorUsingParallel
    {
        public void DoProcess(int processToCreate, int concurrencyLevel)
        {
            SetTurboMode();

            Debug("Creating {0} process definitions", processToCreate.ToString());

            var actions = new Action[processToCreate];
            for (int i = 0; i < processToCreate; i++)
            {
                actions[i] = () => DoSomething(1000);
            }

            Debug("Start queueing and invoking all {0} processes", processToCreate.ToString());
            var options = new ParallelOptions();
            options.MaxDegreeOfParallelism = concurrencyLevel;
            //options.TaskScheduler = new LimitedConcurrencyLevelTaskScheduler(concurrencyLevel); -- we can achieve the same result with this
            Parallel.Invoke(options, actions);

            Debug("All processes have been completed");
        }

        private void DoSomething(int Sleep)
        {
            Debug("Doing something here");
            Thread.Sleep(Sleep);
        }

        /// <summary>
        /// oh i just wish the framework would have this in place like Console.WriteLine
        /// </summary>
        /// <param name="format"></param>
        /// <param name="args"></param>
        private static void Debug(string format, params object[] args)
        {
            System.Diagnostics.Debug.WriteLine(
                string.Format("{0} - Thead#{1} - {2}",
                    DateTime.Now.ToString("dd-MMM-yyyy HH:mm:ss.fff"),
                    Thread.CurrentThread.ManagedThreadId.ToString(),
                    string.Format(format, args)));
        }
        /// <summary>
        /// This is not intended for production purpose
        /// </summary>
        private static void SetTurboMode()
        {
            int t, io;
            ThreadPool.GetMaxThreads(out t, out io);
            Debug("Default Max {0}, I/O: {1}", t, io);

            var success = ThreadPool.SetMinThreads(t, io);
            Debug("Successfully set Min {0}, I/O: {1}", t, io);
        }
    }
}