I reached a milestone this late night with The Gemli Project and decided to go ahead and release it. This release follows an earlier blog post describing some syntactical sugar that was added along with pagination and count support, here:

http://www.jondavis.net/techblog/post/2009/10/05/GemliData-Pagination-and-Counts.aspx

Remember those tests I kept noting that I still need to add to test the deep loading and deep saving functionality? No? Well I remember them. I kept procrastinating them. And as it turned out, I’ve discovered that a lesson has been taught to me several times in the past, and again now while trying to implement a few basic tests, a lesson I never realized was being taught to me until finally just this late night. The lesson goes something like this:

If in your cocky self-confidence you procrastinate the tests of a complex system because it is complex and because you are confident, you are guaranteed to watch them fail when you finally add them.

Mind you, the “system” in context is not terribly complex, but I’ll confess there was tonight a rather ugly snippet of code to wade through, particularly while Jay Leno or some other show was playing in the background to keep me distracted. On that latter note, it wasn’t until I switched to ambient music that I was finally able to fix a bug that I’d spent hours staring at.

This milestone release represents 130 total unit tests since the lifetime of the project (about 10-20 or so new tests, not sure exactly how many), not nearly enough but enough to feel a lot better about what Gemli is shaping in itself. I still yet need to add a lot more tests, but what these tests exposed was a slew of buggy relationship inferences that needed to be taken care of. I felt the urgency to release now rather than continue to add tests first because the previous release was just not suitably functional on the relationships side.

I'm kicking myself because I always just assumed that generic dictionaries in C#, i.e. Dictionary<string, object>, were case-sensitive. I always assumed that in order to have a case-insensitive dictionary, you had to do one of two things:

1. public class CaseInsensitiveDictionary<TValue> : Dictionary<string, TValue> { ... }
2. var item = myDictionary.Find(kvp=>kvp.Key.ToLower() == myKey.ToLower());

Actually, Dictionary<K,V> has a constructor that accepts a comparer object with which you can take advantage of prefab case-insensitive key-matching behavior.

Dictionary<string,object> myDictionary = new Dictionary<string,object>(StringComparer.InvariantCultureIgnoreCase);

I realized this upon stumbling across the MSDN documentation, which, of course, everyone reads for fun. 

http://msdn.microsoft.com/en-us/library/ms132073.aspx

Earlier this late night I posted the first major overhaul update of Gemli since a month ago. This brings a huge number of design fixes and new features to Gemli.Data and makes it even more usable. It's still alpha, though, as there's a lot more testing to be done, not to mention still some new features to be added.

http://gemli.codeplex.com/Release/ProjectReleases.aspx?ReleaseId=33281

From the release notes:

• Added to DataProviderBase: SupportsTransactions { get; } , BeginTransaction(), and BeginTransaction(IsolationLevel)
• Facilitate an application default provider: Gemli.Data.Providers.ProviderDefaults.AppProvider = myDBProvider .. This allows DataModels and DataModelCollections to automatically use the application's data provider when one hasn't already been associated with the model (so you can, for example, invoke .Save() on a new model without setting its provider manually)
• Replace (and drop) TypeConvertor with DbTypeConverter.
• Operator overloads on DataModelQuery; you can now use == instead of IsEqualTo(), but this comes at the cost of chainability. (IsEqualTo() is still there, and still supports chaining.)
• Added SqlDbType attribute property support. DataType attribute value is no longer a DbType and is instead a System.Type. The properties DbType and SqlDbType have been added, and setting any of these three properties will automatically translate and populate the other two.
• DataType attribute value is no longer a DbType and is instead a System.Type. The properties DbType and SqlDbType have been added, and setting any of these three properties will automatically translate and populate the other two.
• WhereMappedColumn is now WhereColumn
• Facilitate optional behavior of inferring from all properties not marked as ignore, rather than the behavior or inferring from none of the unattributed properties unless no properties are attributed. The latter behavior--the old behavior--currently remains the default. The new behavior can be applied with DataModelTableMappingAttribute's PropertyLoadBehavior property
  • This may change to where the default is InferProperties.NotIgnored (the new behavior), still pondering on this.
• Add sorting to MemoryDataProvider
• TempMappingTable is now DataModelMap.RuntimeMappingTable
• ForeignDataModel.ForeignXXXX was backwards and is now RelatedXXXX (multiple members)
• DataModelFieldMappingAttribute is now DataModelColumnAttribute.
• DataModelMap.FieldMappings is now DataModelMap.ColumnMappings.
• DataModelTableMappingAttribute is now DataModelTableAttribute.
• DataModelForeignKeyAttribute is now ForeignKeyAttribute.
• The XML serialization of all elements now follows these renamings and is also now camelCased instead of ProperCase.
• XML loading of mapping configuration is getting close, if it isn't already there. Need to test.
• Added DataModelMap.LoadMappings(filePath)

I was sitting at my Mac Mini to remote into my Windows 7 machine and I had this MonoDevelop 2.0 icon sitting there (I had installed it a couple days ago just to see it install and run) and I thought I should give Gemli.Data a quick, simple test run.

(Click to view)

Notice the “Application output” window. It werkie!!

I was concerned that Mono hadn’t been flushed out well enough yet to be stable with Gemli.Data, I do a lot of reflection-heavy stuff with it to get it to infer things, not to mention System.Data dependency stuff. But it seems to work fine.

This isn’t by any means a real test, more of a quick and dirty mini smoke test. I was very happy to see that it works fine, and my confidence in Mono just went up a notch.

Not shown in the screen shot, I expanded it a little tiny bit to have multiple records, then I added sorting in the query. Apparently Gemli doesn't have sort support in the MemoryDataProvider yet so I used LINQ-to-Objects. Yes, LINQ syntax worked in Mono, too! Nifty!

I’ve just open-sourced my latest pet project. This project was started at around the turn of the year, and its scope has been evolving over the last several months. I started out planning on a framework for rapid development and management of a certain type of web app, but the first hurdle as with anything was picking a favorite DAL methodology. I tend to lean towards O/RMs as I hate manually managed DAL code, but I didn’t want to license anything (it’d mean sublicensing if I redistribute what I produce), I have some issues with LINQ-to-SQL and LINQ-to-Entities, I find nHibernate difficult to swallow, I have some reservations about SubSonic, and I just don’t have time nor interest to keep perusing the many others. Overall, my biggest complaint with all the O/RM offerings, including Microsoft’s, is that they’re too serious. I wanted something lightweight that I don’t have to think about much when building apps.

So as a project in itself I decided first to roll my own O/RM, in my own ideal flavor. Introducing Gemli.Data! It’s a lightweight O/RM that infers as much as possible using reflection, assumes the most basic database storage scenarios, and helps me keep things as simple as possible.

That’s hype-speak to say that this is NOT a “serious O/RM”, it’s intended more for from-scratch prototype projects for those of us who begin with C# classes and want to persist them, and don’t want to fuss with the database too much.

I got the code functioning well enough (currently 92 unit tests, all passing) that I felt it was worth it to go ahead and let other people start playing with it. Here it is!

Gemli Project Home: http://www.gemli-project.org/ 

Gemli Project Code: http://gemli.codeplex.com/

Gemli.Data is currently primarily a reflection-based mapping solution. Here’s a tidbit sample of functioning Gemli.Data code (this comes from the CodePlex home page for the project):

// attributes only used where the schema is not inferred
// inferred: [DataModelTableMapping(Schema = "dbo", Table = "SamplePoco")]
public class SamplePoco
{
    // inferred: [DataModelFieldMapping(ColumnName = "ID", IsPrimaryKey = true, IsIdentity = true, 
    //     IsNullable = false, DataType = DbType.Int32)] // note: DbType.Int32 is SQL type: int
    public int ID { get; set; }

    // inferred: [DataModelFieldMapping(ColumnName = "SampleStringValue", IsNullable = true, 
    //     DataType = DbType.String)] // note: DbType.String is SQL type: nvarchar
    public string SampleStringValue { get; set; }

    // inferred: [DataModelFieldMapping(ColumnName = "SampleDecimalValue", IsNullable = true, 
    //     DataType = DbType.Decimal)] // note: DbType.Decimal is SQL type: money
    public decimal? SampleDecimalValue { get; set; }
}

[TestMethod]
public void CreateAndDeleteEntityTest()
{
    var sqlFactory = System.Data.SqlClient.SqlClientFactory.Instance;
    var dbProvider = new DbDataProvider(sqlFactory, TestSqlConnectionString);

    // create my poco
    var poco = new SamplePoco { SampleStringValue = "abc" };

    // wrap and auto-inspect my poco
    var dew = new DataModel<SamplePoco>(poco); // data entity wrapper

    // save my poco
    dew.DataProvider = dbProvider;
    dew.Save(); // auto-synchronizes ID
    // or...
    //dbProvider.SaveModel(dew);
    //dew.SynchronizeFields(SyncTo.ClrMembers); // manually sync ID

    // now let's load it and validate that it was saved
    var mySampleQuery = DataModel<SamplePoco>.NewQuery()
        .WhereProperty["ID"].IsEqualTo(poco.ID); // poco.ID was inferred as IsIdentity so we auto-returned it on Save()
    var data = dbProvider.LoadModel(mySampleQuery);
    Assert.IsNotNull(data); // success!

    // by the way, you can go back to the POCO type, too
    SamplePoco poco2 = data.Entity; // no typecast nor "as" statement
    Assert.IsNotNull(poco2);
    Assert.IsTrue(poco2.ID > 0);
    Assert.IsTrue(poco2.SampleStringValue == "abc");

    // test passed, let's delete the test record
    data.MarkDeleted = true; 
    data.Save();

    // ... and make sure that it has been deleted
    data = dbProvider.LoadModel(mySampleQuery);
    Assert.IsNull(data);

}

Gemli.Data supports strongly typed collections and multiple records, too, of course.

var mySampleQuery = DataModel<SamplePoco>.NewQuery()
    .WhereMappedColumn["SampleStringValue"].IsLike("%bc");
var models = dbProvider.LoadModels(mySampleQuery);
SamplePoco theFirstSamplePocoEntity = models.Unwrap<SamplePoco>()[0];
// or.. SamplePoco theFirstSamplePocoEntity = models[0].Entity;

Anyway, go to the URLs above to look at more of this. It will be continually evolving.

I've posted a mini-subproject:

http://www.CAPTCHAThisYo.com/

The site is self-explanatory. The idea is simple. I want CAPTCHA. I don't want to support CAPTCHA in my apps. I just want to drop in a one-liner snippet somewhere and call it done. I think other people share the same desire. So I now support CAPTCHA as a CAPTCHA app. I did all the work for myself so that I don't have to do that work. I went through all that trouble so that I don't have to go through the trouble .... Wait, ...

Seriously, it's not typical CAPTCHA, and it's Not Quite Done Yet (TM). It's something that'll evolve. Right now there isn't even any hard-to-read graphic CAPTCHA.

But what I'd like to do is have an ever-growing CAPTCHA questions library and, by default, have it just rotate through them randomly. The questions might range from shape detection to riddles to basic math. I'd really like to have some kind of community uploads thingmajig with ratings, so that people can basically share their own CAPTCHA solutions and they all run inside the same captchathisyo.com CAPTCHA engine. I'm just not sure yet how to pull that off.

Theoretically, I could take the approach Microsoft took when C# was initially released (long, long ago, in a galaxy far, far away), they had a cool insect sandbox game where you could write a .NET class that implements some interface and then send it up to the server and it would just run as an insect on the server. The objective of the game was to write the biggest killer/eater. I'm not sure how feasible the idea is of opening up all .NET uploads to the server, but it's something I'm pondering on.

Anyway, the concept has been prototyped and the prototype has been deployed is sound, but I still need to work out cross-site scripting limitations, bear with me. I still need to find a designer to make something beautful out of it. That said, feel free to use it and give feedback. Stand by.

About a week and a half ago I finally started breaking ground (that is, created a Visual Studio soluton and started writing code, with the intention of it being used in production) on a big composite pet project I've been considering for a very long time. I won't get into what the project is going to be about (at least not now), but I will say parts of it are multi-faceted software foundation / framework. So, once I'm done, I'll be able to take a bunch of the code I wrote here and re-apply it on a completely different project. This is important to me as I am seriously in need of gaining some inertia in pet project development and deployments.

So right away I started writing tests to "prove out" each little piece of code as I wrote it. I'd heard the TDD gospel and wanted to convert, but I don't have a lot of TDD practice under my belt. But right away I had about 75% code coverage, which I felt was pretty good compared to 0% of projects in the past. Every time I wrote a little bit of code, I hit Ctrl+R, A to be sure nothing I just did failed. This worked fine, at the beginning, while working with simple, lightweight objects.

Pretty soon I found myself implementing abstract provider/service classes, and then create at least one or two implementations of them. And then I wrote this big dependency object that gets passed around back and forth. By the time I had some prototype code written, I realized that my tests would not break because I hadn't been writing code around my tests, I had been writing my tests around the code I had been writing. And at this point I was clueless as to whether the code I had just written would work, because this was all plumbing code. I wanted to at least try something and see if the basics were working, so I created a console application and wrote some code with Console.Write()'s to get a visual on the output. But at this point things were getting messy. My dependency object I had created was created for the one implementation class of my abstract provider/service object. I now had a throw-away console app in my solution that didn't belong. And my test coverage was down to something like 5%.

That's when I realized I was going about this all wrong. For months I had a hard time understanding why TDD proponents argued that writing code before their tests is "putting the cart before the horse", when writing tests before code seemed so backwards to me. But now it started to click. For many years, I've been designing code in my head, then implementing those designs in code, and if it works, great, if not, I fix the bugs. I'm starting to see now how I need to retrain myself. It's okay to design code in your head. You just need to document those designs in tests. At first, they won't even compile. Fine. But at least document your designs in tests.

Funny, I haven't ever heard anyone make the connection, but writing tests with mocks of the way you want your code to work is exactly the same to a programmer as wireframes are to web design. One should not design a web page by throwing some HTML into a web site, trying to make it look nice and complete, and then showing it to the customer for the first time and saying, "Here you go, it's done. Now tell me what's wrong with it." Rather, a professional web production team would first collect the requirements (in workflow verbiage) of the web site, throw together some wireframe mock-ups, create Photoshop design comps in parallel, and then go to implementation, all the while following up with the customer for feedback during each and every step along the way. TDD is exactly the same but for the programmer. The tests are wireframes and business rules, just like a web application spec has bullet-point business rule text that serves as manual tests for all parties involved to both define and validate the implementation.

Even if there is no third party customer who would approve your "wireframes" and you're working solo, the analogy of "wireframing a web design" should still apply. A web designer/engineer can, but shouldn't, create a web site before wireframing it. He should start with a napkin, and go from there. Likewise, one can, but shouldn't, write code before writing tests for that code, because tests are not just validations that assumed functionality works but are also definitions of assumed functionality, and without definitions of assumed functionality there are really no coding objectives.

The hardest part for me to do TDD so far in this new experience is realizing I wasn't doing TDD in the first place, and then going back and scrapping all the code I had just written, whether it was good or not. I knew it had a little bit of 'bad' mixed in with the good, and that should not be acceptable. Awareness of "a little bit of bad code buried in there" typically means mostly bad code in the long run, and a lot of wasted time, because bad code cascades and affects everything else, like yeast.

One other thing about these incidents is that I've also been re-learning the theory of YAGNI (you ain't gonna need it!). I was getting too comfortable with the idea of introducing extra method signatures on objects that were not getting properly tested to begin with, and for which my use cases were purely theoretical, and not truely intentional. I've argued here in this blog in the past that YAGNI is extremely naive and not the appropriate attitude when writing software in smaller shops (like one or two man bands) because the multi-role developer often knows exacty what he needs and should not be withheld from adding what he needs in order to do his job. That's great, ignore YAGNI, so long as you're taking an "undisciplined", less-formal approach to software or are writing software that is not intended to be reused such as a framework. However, in my case, I'm writing framework bits, and I must balance ease of use by way of high versatility versus ease of use by way of simplicity. Other programmers, including myself at some later point in time, should not be confused with options. Programmers are software users, just like their end-users are software users. So the end-users use the mouse and click simple things, whereas programmers use framework code to write code, fine, either way, they're all users. The Apple and Google approaches are to keep it simple, stupid (KISS). Don't overwhelm the user with options, especially if either of two options reach the same outcome. I should define one "best practices" path and only introduce the other path when it is needed, not when it is slightly and occasionally convenient.

Part of the reason why I write like such is to drive it into my head. I still settle for less than what I preach sometimes.

There's one other thing; tests are a part of the TDD strategy, but I'm also beginning to think that once I have made some headway into these framework bits to be able to actually write some code that passes tests, I might also start writing some application/implementation project code. Call it a manual test if you like. The tests themselves should not lose priority. But I think it makes sense, to an extent, to see code written in action as soon as it's feasible in order to prove out code design prototypes. Writing application code should NOT actually be additional tests but rather take advantage of what's alreay passing and help guide the tests as the tests are being written. In this light, the only problem I have with TDD is that TDD is spoken of too often in terms of unit and integration tests, whereas the whole lifecycle of designing with tests and of testing must be much broader than unit and intgration tests. TDD starts with unit tests, but unit tests, by definition, only test one tiny unit of functionality. In order to really drive your design and development through tests, you need to be able to test workflows, too. Starting from Point A, given n parameters, we should ultimately arrive at B, and this might involve several steps.

Oh, I still need to watch Rob Conery's MVC Storefront videos where he, too, learned TDD. I got about ten or so videos in until I caught up, and then stopped watching while he kept streaming them out.

C# 3.0 has a cool feature I didn’t realize existed until now. I already knew that you can create generic methods, like so:

protected static T CaseInsensitiveFindValue<T>(Dictionary<string, T> dic, string key)
{
    if (dic.ContainsKey(key)) return dic[key];
    foreach (var kvp in dic)
    {
        if (kvp.Key.ToLower() == key.ToLower()) return kvp.Value;
    }
    return default(T);
}

That goes back to 2.0. What I didn’t realize until now was that when you invoke this method, you don’t have to pass the type in as a type argument for T. The type can be inferred!

var value = CaseInsensitiveFindValue(myDictionary, myKey);

Here’s my idea of simple dependency mapping, without an IoC container. The idea is to drop your binaries into the bin directory, optionally (yes, optionally) name your dependencies somewhere such as in app.config / web.config, and let ‘er rip.

I still hate Castle Windsor, et al, and will never write code around it if I can avoid it.

By the way, this is just DI, not IoC. For IoC without an IoC container, I still like eventing. See my past posts:

• IoC By Way Of Simple Event Raising: Part 1
• IoC By Way Of Simple Event Raising: Part 2 - Contained Interfaces and Evented Methods

Dependency mapping implementation example (conceptual; one can and should use better code than this):

// Scenario infers IMyService as a referenced/shared interface-only type library
// used on both the invoker and the IMyService implementor.

// Example 1:
// match on Type.FullName
IMyService myService = (IMyService)Activator.CreateInstance(
	Dependency.GetDependency<IMyService>(ConfigurationManager.AppSettings["myServiceProvider"]));
myConsumer.MyDependency = myService;

// Example 2:
// match on Type.Name case insensitively; NOTE: This is not recommended.
IMyService myOtherService = (IMyService)Activator.CreateInstance(
	Dependency.GetDependency<IMyService>(ConfigurationManager.AppSettings["myServiceProvider2"], true));
myConsumer.MyDependency = myOtherService;

// Example 3:
// Only one match? Too lazy to name it? Just take it. NOTE: This is not recommended.
IMyService whateverService = (IMyService)Activator.CreateInstance(
	Dependency.GetDependency<IMyService>());
myConsumer.MyDependency = whateverService;

Code:

using System.Reflection; //, etc...

// ...

public class Dependency
{
    public static Dictionary<Type, List<Type>> KnownServices
        = new Dictionary<Type, List<Type>>();


    /// <summary>
    /// Returns any <see cref="Type"/> in the current <see cref="AppDomain"/>
    /// that is a <typeparamref name="T">T</typeparamref>. 
    /// </summary>
    /// <typeparam name="T">The type in the <see cref="AppDomain"/> that the
    /// return <see cref="Type"/> should be or inherit.</typeparam>
    /// <returns></returns>
    public static object GetDependency<T>()
    {
        return GetDependency<T>(null);
    }

    /// <summary>
    /// Returns any <see cref="Type"/> in the current <see cref="AppDomain"/>
    /// that is a <typeparamref name="T">T</typeparamref>. 
    /// If <paramref name="by_name"/> is not <code>null</code>, only the type(s) matching
    /// the specified <paramref name="by_name"/> with the <see cref="Type.FullName"/> is returned.
    /// </summary>
    /// <typeparam name="T">The type in the <see cref="AppDomain"/> that the
    /// return <see cref="Type"/> should be or inherit.</typeparam>
    /// <param name="by_name">The type name that should match the return value(s).</param>
    /// <example>object myService = Activator.CreateInstance(GetDependency&lt;IMyService&gt;);</example>
    public static object GetDependency<T>(string by_name)
    {
        return GetDependency<T>(by_name, false);
    }

    /// <summary>
    /// Returns any <see cref="Type"/> in the current <see cref="AppDomain"/>
    /// that is a <typeparamref name="T">T</typeparamref>. 
    /// If <paramref name="by_name"/> is not <code>null</code>, only the type(s) matching
    /// the specified name will be returned.
    /// </summary>
    /// <typeparam name="T">The type in the <see cref="AppDomain"/> that the
    /// return <see cref="Type"/> should be or inherit.</typeparam>
    /// <param name="by_name">The type name that should match the return value(s).</param>
    /// <param name="short_name_case_insensitive">If true, ignores the namespace,
    /// casing, and assembly name. For example, a match on type <code>Dog</code>
    /// might return both <code>namespace_a.doG</code> and <code>NamespaceB.Dog</code>.
    /// Otherwise, a match is made only if the <see cref="Type.FullName"/> matches
    /// exactly.
    /// </param>
    /// <returns>A <see cref="Type"/>.</returns>
    /// <example>object myService = Activator.CreateInstance(GetDependency&lt;IMyService&gt;);</example>
    public static object GetDependency<T>(string by_name, bool short_name_case_insensitive)
    {
        if (by_name != null && !short_name_case_insensitive)
        {
            return Type.GetType(by_name, true);
        }
        Init<T>();
        var t_svcs = KnownServices[typeof(T)];
        if (string.IsNullOrEmpty(by_name))
        {
            if (t_svcs.Count == 0) return null;
            if (t_svcs.Count == 1) return t_svcs[0];
            return t_svcs; // more than one, return the whole list
        }
        return t_svcs.Find(t => t.Name.ToLower() == by_name.ToLower());
    }

    private static readonly Dictionary<Type, bool> Inited = new Dictionary<Type, bool>();
    private static void Init<T>()
    {
        if (Inited.ContainsKey(typeof(T)) && Inited[typeof(T)]) return;
        if (!KnownServices.ContainsKey(typeof(T)))
            KnownServices.Add(typeof(T), new List<Type>());
        var refAssemblies = new List<Assembly>(AppDomain.CurrentDomain.GetAssemblies());
        foreach (var assembly in refAssemblies)
        {
            Type[] types = assembly.GetTypes();
            foreach (var type in types)
            {
                if (type.IsClass && !type.IsAbstract &&
                    typeof(T).IsAssignableFrom(type))
                {
                    KnownServices[typeof(T)].Add(type);
                }
            }
        }
        Inited[typeof(T)] = true;
    }
}

I created a new WCF demo for EntitySpaces, one of the most popular ORM solutions available for .NET which now comes with its own code generator (no longer relies on CodeSmith or myGeneration). The demo is bundled in the Release Candidate for v2009 Q1. (The developer version is released, trial version will be released tomorrow.) This one includes both console and Windows Forms clients, and a console-based service, for showing the barebones basics of what it takes to get EntitySpaces working with WCF. Both full proxies (EntitySpaces runtime libraries referenced on the client) and lightweight proxies/stubs (*no* EntitySpaces runtime libraries referenced on the client) are demonstrated, but the lightweight demo is currently limited to a console app.

Next on my plate will be a WPF demo for the lightweight proxies/stubs. No guarantees...

Anyway, here’s the documentation that went with the demo. It got posted on the EntitySpaces blog.

http://www.entityspaces.net/blog/2009/01/25/EntitySpaces+2009+And+WCF.aspx