Wednesday, August 13, 2014

Yield Return

Introduction

You're all familiar with yield return by now right? You know, that yellow traffic sign that means you have to yield to oncoming traffic, then return from whence you spewed forth never to return. Wait, what's that you say? This is a .Net coding blog not a traffic blog? Oh right, so maybe I meant the yield return statement within C#. I'll admit that while I had heard of yield return before this week, and I had even read a little bit about it online, I didn't really understand what it was and what it was for. I don't know about you, but if I start to read about something and I can't get a feel for why I would use it, I care less about how to use it. So, in addition to showing you what yield return does and how to use it, I'll cover a useful scenario where you might want to use it.

What it Be

yield return <x> lets you return an enumerable type from a function call, 1 element at a time. If you're wondering how it lets you do so 1 at a time without losing it's place the next time you call, it's because .Net keeps track of your place so the next time you call the method within an iteration loop, it picks back up right after where you left off with the previous yield return statement. Here's a quick contrived example to help explain:

using System;
using System.Collections.Generic;

namespace BlogYield
{
    class Program
    {
        public static IEnumerable<int> GetRandoms(int count, int min, int max)
        {
            Random rnd = new Random();
            for (int index = 0; index < count; index++)
            {
                int random = rnd.Next(min, max + 1);
                yield return random;
            }
        }

        static void Main(string[] args)
        {
            foreach (int random in GetRandoms(10, 0, 100))
                Console.WriteLine(random);
            Console.ReadKey();
        }
    }
}



This is a full console application in C# that displays 10 random numbers in the range 0 to 100. The function GetRandoms returns an IEnumerable of type int, and proceeds to loop through count times producing random numbers. Notice that neat little bit of yield return w/in the loop? This means that the first time you execute the function, it will go through all the lines of the function in sequence; create the Random object, start the for loop, create the random number, return the first random number. Now the second time the function is called (within the loop in the Main function), the loop within GetRandoms continues on to the next iteration, a new random integer is set, and yield return is called with the next item. Nifty huh? I know there are better ways to accomplish this, a single for-loop being foremost in my mind, but this really does have some good uses. I promise! Let's look at one now that you know the mechanics.

Use Case: Merging Lists

Merging multiple lists that contain the same type can be a bit of a chore. In drops yield return to the rescue! Have a look at this example:

        public static IEnumerable<T> MergeLists<T>(params IEnumerable<T>[] lists)
        {
            foreach (var list in lists)
            {
                foreach (var item in list)
                    yield return item;
            }
        }

//...
//now we call it...
            var list1 = new List<int>();
            list1.Add(2);
            list1.Add(4);
            var list2 = new List<int>();
            list2.Add(0);
            list2.Add(100);
            foreach (var number in MergeLists<int>(list1, list2))
                Console.WriteLine(number);




As you can imagine, our output is 2, 4, 0, 100 (on separate lines). And, thanks to the magic of generic methods (I suppose I'll have to cover those in the blog at some point) and params, we can pass in any number of lists of any type, so long as they're all the same type and all the "lists" are enumerable. Pretty freakin sweet! Your original lists remain untouched, life is good.

[A quick note: I got this idea from here and just modified it a little to fit my own purposes. Thank you John K!]

What's Next?

Find some other uses for yield return and practice using it. I'm sure you'll think of something if you try. Or, another good exercise might be to try to recreate the above MergeLists method on your own. Even if you don't use a generic method or the params keyword, it will still help cement your understanding of yield return. Lastly, check the resources at the bottom of the article, especially the yield C# reference. They go into a bit more detail about how yield return works than I did, as I just glossed over it so I could get to a practical example quickly.

Resources

A Practical Use of "yield" Keyword in C#
yield (C# Reference)

Thursday, August 7, 2014

Website Optimization: Bundling and Minification

Intro

Website optimization is a goal of the highest order for web developers, and if it's not for you it should be. Why? Users don't like to wait. If a user waits more than a few seconds for a website, chances are good they'll go to a different website. You don't want that to happen to your websites do you? I'll assume not. If you don't like users then I suppose you do want this to happen, but I can't please everyone. Hey, just ignore the advice if that's you!

There are quite a plethora of ways to optimize websites. The server side code could be slow, the database behind it (if there is one) could be slow, internet connections can be problematic, javascript on your pages can be pokey, and you could just be forcing users to download too much stuff when they hit your site. This article concerns just a portion of the last item there, too much crap to download. Lots of things can cause your site to have a bloated download size or too many files to download; too many images, large images, uncompressed static files, server-side controls that render bloated html content to the client, too many and/or too large scripts and css files and more. Once again, this last item is what we're dealing with.

I hope you have a good sense from the above section that there can be quite a number of reasons for a slow site. Keep in mind we're covering just one of them here! If you're developing websites, I strongly encourage you to do some more research on teh interwebs to see what all else you can do to speed up your websites.

What is Bundling?

Bundling is pretty much what it sounds like; you bundle things together. In this case, I'm talking about bundling together javascript files and css files so that there are fewer hits to your server. Put on your pretending cap for a moment and picture a world where your website has just a single .js file. A user who visits your site will only download that 1 js file, so you've cut down on their outbound connections and your inbound connections. FTW! Now picture in yer noggin what that js file would look like if you have anything larger than a basic site: it would be a jumbled mess and it would be extremely difficult to find anything. That there fill will be huge! This is where bundling comes in. ASP.Net bundling lets you, with server-side code, specify group(s) of javascript or css files to bundle together at runtime. This means that you can keep them files separated (thanks Offspring!) in the development environment so your site is better organized and your code easier to manage, but your users get served much fewer file(s) when they view your page.

What is Minification

Minification is a podunk form of compression. In the web world, it means you strip js and css files down to a bare minimum of text that still functions the same. Take for example this line of js:

var myVariable = 3;

It doesn't seem like much, but this line is 19 characters long. Compare it to this line:

var m=3;

This new line is 8 characters long, and means the same thing. The variable name is different but that's it. Now going back and making all your js look like this will probably get you punched by your co-workers. Instead, let ASP.Net handle it for you! You can keep your js code looking nice and pretty on the development side, while serving your users much more compressed-looking code that works exactly the same.


Code!

Let's see how to go about using the built-in bundling and minification in .Net. Fire up VS Express and create a new ASP.Net Web Application. Mine is named blogbundlemin.
 

Pick the MVC template and change authentication to "No Authentication":


After a few seconds you should now have yourself a basic website template. Expand the scripts folder that VS made for you in your website and create a few js files. It doesn't matter what you put in them, but here is what I ended up with (zz_1.js, zz_2.js and zz_3.js):

The contents are pretty slim too:

zz_1.js:
var myVariable1 = 1234;

zz_2.js:
var myVariable2 = "purple";

zz_3.js:
var myVariable3 = "something or other";


Now open up the App_Start folder. In there is a special file that ASP.Net has already generated for you called BundleConfig.cs. Open up that file. You'll see that there's already a method in this class called RegisterBundles(), and there's some code in there already that is adding bundles. Go ahead and drop some code in there at the top of the method:

            bundles.Add(new ScriptBundle("~/bundles/local").Include(
                "~/Scripts/zz_*"));


This tells .Net to create a bundled named "~/bundles/local", and include in this bundle all the files named "~/Scripts/zz_*".

Great! Now we have a cool bundle, but how do we use it in one of the pages of our site? Open up Views/Shared/_Layout.cshtml. For those not familiar with MVC, this is like a master page from the world of web forms. Most if not all pages in the site will use this _Layout page to use a common layout and theme throughout the site. Towards the bottom of the file you can see a couple of lines that look like @Scripts.Render; add one for your own bundle. Here's what your code should look like:

    @Scripts.Render("~/bundles/jquery")
    @Scripts.Render("~/bundles/bootstrap")
    @Scripts.Render("~/bundles/local")
    @RenderSection("scripts", required: false)


Now run the project in Internet Explorer. Once the site finished loading, hit F12 to bring up developer tools. Now click on the weird little "network" icon on the left then click the Play button.


Now hit Ctrl-F5 to force IE to refresh all its local content for this page, then click the stop button. Your developer tools should look like this:



You can see your lovely bundle in there! If you double-click this line then click the "Response body" link you'll see what's in the bundle:


Yes that's right ladies and germs, .Net has both minified your content (it put all 3 variable declarations on one line, got rid of the trailing semicolon, and removed unnecessary whitespace)  as well as bundling all 3 files together. Nifty!

What's Next?

You can bundle css files too. It's just as easy as bundling js files. Give it a whirl!

Can you figure out how to turn on and off bundling? Hint: there are at least 2 good ways to do it.

You can also go super-bold and explore other ways to optimize websites. Heck, just play around with the IE developer tools some more if you're bored, they're fun.

Resources

ASP.Net Bundling and Minification

Thursday, July 31, 2014

Configure TFS (web version) for builds.

Yay! You've gotten your TFS (web version) installed and your code checked in.  Time to automate your builds so when you check in your code the phrase "works on your machine" will apply to "works on that web machine" as well :).

To configure TFS for builds:

Step 1: open up VS2013 Express for Web and open your project in question that's been checked in:

Step 2: go to the Team Explorer tab and click on Builds.
Step 3: Click on Builds, you should see the following screen:
Step 4: Click on New Build Definition:

Step 5: options! options! options!  Want to change your build name to something pertinent?  Go fer it!  For my purposes every time I check in some code I want it to build to verify what I wrote is compatible with what's checked in.
     a)  For this i'm clicking on the Trigger menu item and selecting Continuous Integration as the option for me.
     b)  Click on Source Settings:   If you wish to exclude certain paths in your code for building you can do that here. The active paths listed will work for my purposes.
     c)  Click on Build Defaults:  needing to build out your files to a particular location?  Here's where you can do that for your project.  I don't require that for my project so I'm going with the top option.
Step 6:  I'm going to rename my build process so I click on General and rename my build definition name.  I'll use "Build On Check In" for a meaningful name for keeping up with my project management.  Now save the Build Definition.
Now you should see on the right within Team Explorer under All Build Definitions the name of your project, for my purposes "Build On Check In".  Right click on it and choose "Queue New Build".  You should see a screenshot like the following:
Step 7:  You should see a screen like the following screenshot.  For my project purposes all options listed will work well for my project.  If you need additional parameters for a more specialized build they can be entered on the Parameters tab.
Step 8:  click Queue when you're ready to send up your new build definition to the server.  Huzzah! You should now have a build definition.
     a)  Log into your *visualstudio.com website to verify for yourself, here's the one I'd just created:
     b)  Click on Build, you should see your new build option on the lower left hand side.

Time to verify the build works, aka I get to break code on purpose!  Compiled my code and received the error expected, lets see what happens when I check in knowingly bad source code:
When I go to the web tab with my visualstudio account info, error!
Huzzah!  Fix your code and check it back in.  You should see a successful build like:
Yay, configuration o' builds complete.





Friday, July 25, 2014

Install TFS Web Based Version + Check in some code!

TFS (Team Foundation Server) vs TFS (Team Foundation Service).  Micro$oft, WTH?  Naming conventions aside, you now have options for project management:  Server (you get the pleasure of hosting) or Service (to the cloud!)

I could come up with clever reasons to use TFS for home, or let Google (yeah the irony) come up with reasons: http://blogs.msdn.com/b/msgulfcommunity/archive/2011/01/25/10-reasons-to-deploy-team-foundation-server.aspx.  Or regale you of tales of 9+ hours trying to configure the desktop version of TFS for local usage: aka: Tidwell not approved!

TFS as Team Foundation Service:
Go to http://www.visualstudio.com/products/visual-studio-online-user-plans-vs to create a MS account if you don't have one already. 

to do so click on Create Account Now:
At this step you'll need to name your account.  For personal use i'm going with dtidwelltfs which will work out to dtidwelltfs.visualstudio.com

then you'll get to the main page to manage your projects within Team Foundation Service:
Time to create a new project!  I'll use the generic ChessBlog (a project Peter had used in the past for a different blog):
after clicking Create Project:
Huzzah!  Click Open with Visual Studio to connect to get to your source code for binding.  You'll get the typical Launch App popup:
I happen to have 2 versions of Visual Studio 2013 installed: VS2k13 for Desktop and VS2k13 for Web.  This project is web based, I'll open for Web.
  
Double Huzzah!

Microsoft requires you to be logged into your VS account, get ready for yet another popup:
And another huzzah for good measure.  Within VS2k13 for Web you should hopefully have a Team Explorer tab available to you like the following:
I've filled in my local path to my source code and click Map & Get.  You should see the following after doing so:
Yeah buddy!  Mapping complete, time to send your code to the cloud.  Hopefully Source Control Explorer is open, if not open it.  You should be able to add items in your folder via right clicking and selecting Add items to folder, then selecting/deselecting any coding items you want part of your TFS project.
After getting your code into the project, right click and choose Check In Pending Changes.  After selecting that you should see:
Success!  At this point you might be wondering: It's showing up in the explorer window but is it really there?  Go back to your TFS webpage and you should see something like the following:
Click on your project (BlogChess) and you should see:

That should cover the basics of installing TFServices.  Tune in next time when we configure TFServices for bulids/tests!



Thursday, July 24, 2014

NoSQL - MongoDB; Querying Data

Intro

In Part 1 of the series on NoSQL we were introduced to some of the general concepts behind this movement and how some of the technologies work in a general sense. In Part 2 we setup a MongoDB instance and wrote some C# code to insert data into the DB. Here in the final installment of the series we'll cover getting our data back out of the MongoDB instance.

To the Code!

Let's jump on into some code. Open back up the solution file you created back in Part 2, BlogMongo. If you named your stuff the same I did, you can open up default.aspx and default.aspx.cs also as that's where we'll put our code. I'm adding a new button to the form called btnQuery and giving it a new click event, which I'll leave empty for the moment. Now, do you remember this function from last week?

        private MongoCollection GetMongoCollection()
        {
            var client = new MongoClient("mongodb://localhost");
            var server = client.GetServer();
            var database = server.GetDatabase("BlogMongo");
            return database.GetCollection<Stuff>("stuffses");
        }


This is the function we used last week to connect to our database and our collection (table), using the Mongo C# driver. Now let's pull some data out of it; you'll be surprised just how easy this is:

        
        private void QueryStuff()
        {
            var collection = GetMongoCollection();
            var singleQuery = Query<Stuff>.EQ(e => e.SomeInt, 3);
            var stuff = collection.FindOneAs<Stuff>(singleQuery);
            var serializer = new JavaScriptSerializer();
            Response.Write(serializer.Serialize(stuff));
        }


This method pulls a single document/object out of the database and collection and spits it out in the response. The first line of the method just opens up a connection to the database and retrieves a reference to our collection. The second line creates a Mongo query object (you'll have to add a using statement to MongoDB.Driver.Builders). The query object should return object(s) of type Stuff, and the query shall look for an item with a value of 3 for the property SomeInt. The 3rd line tells the collection to pull back a single item of type Stuff that matches the query we created in the previous line. Line 4 creates a javascript serialization object that we're just using to display some friendly output, and line 5 spits out our serialized JSON object to the browser. Yes that's right, 3 lines of code (which we could easily condense further) pulls a specific object out of the database for us. Nifty! Go ahead and call this new method in the click event of your query button so you can see the output. Just in case you're not coding along with me, here's what the output looks like:

{"Id":"d639cae8-906a-4d7c-ab6d-553319a8a70e","SomeInt":3,"SomeString":"yeah, a string","ListOfStrings":["0","1","2"]}

Ok that was some really cool stuff. Almost as cool as an iceberg hitting you in the face! But how do we create some niftier, more complicated queries? Well as it turns out, the Mongo C# driver supports quite a few LINQ operations for data querying so you can have some real fun here! Let's see another example of querying MongoDB data, this time with LINQ:

        private void QueryStuff()
        {
            var collection = GetMongoCollection();
            var singleQuery = Query<Stuff>.EQ(e => e.SomeInt, 3);
            var stuff = collection.FindOneAs<Stuff>(singleQuery);
            var serializer = new JavaScriptSerializer();
            Response.Write(serializer.Serialize(stuff));
            var linqQuery = from item in collection.AsQueryable<Stuff>()
                            where item.SomeString == "yeah, a string"
                            select item;
            foreach (var item in linqQuery)
                Response.Write(serializer.Serialize(item));
        }



Our new code starts on line 6, beneath the prior junk. Line 6 sets up our LINQ query. I'm assuming you're familiar enough with LINQ to either create LINQ queries or google around to figure out how to make them so I won't go into detail on it, but as you can see this query is meant to select all the items from our collection who have a SomeString property with the value "yeah, a string". (Note: you'll need to add 2 more using statements. MongoDB.Driver.Linq and System.Linq). The foreach below it is just our output generation code. Easy, simple, and powerful, that's the best kind of code there is! Here is the new output (note that it includes the prior line of output too, so there's 1 extra object of type Stuff here):

{"Id":"d639cae8-906a-4d7c-ab6d-553319a8a70e","SomeInt":3,"SomeString":"yeah, a string","ListOfStrings":["0","1","2"]}{"Id":"d639cae8-906a-4d7c-ab6d-553319a8a70e","SomeInt":3,"SomeString":"yeah, a string","ListOfStrings":["0","1","2"]}{"Id":"494be80b-5f1a-48c4-9a15-3c01f89d2216","SomeInt":4,"SomeString":"yeah, a string","ListOfStrings":["0","1","2","3"]}{"Id":"a089f1bd-c33b-43ae-9f25-894e0c5e31c2","SomeInt":5,"SomeString":"yeah, a string","ListOfStrings":["0","1","2","3","4"]}{"Id":"097c8d4f-fff4-4daa-a535-75d38fe87199","SomeInt":3,"SomeString":"yeah, a string","ListOfStrings":["0","1","2"]}

That's pretty much all we're going to cover folks! These are just the basics of querying a MongoDB instance, but this will get you pretty far. Here's the complete code file in case you need it:

using MongoDB.Bson;
using MongoDB.Driver;
using MongoDB.Driver.Builders;
using System;
using System.Collections.Generic;
using System.Web.Script.Serialization;
using System.Linq;
using MongoDB.Driver.Linq;

namespace BlogMongo
{
    public class Stuff
    {
        public Guid Id { get; set; }
        public int SomeInt { get; set; }
        public string SomeString { get; set; }
        public IList<string> ListOfStrings { get; set; }
        public Stuff()
        {
            ListOfStrings = new List<string>();
        }
    }

    public partial class Default : System.Web.UI.Page
    {
        private Stuff GenerateStuff()
        {
            var stuff = new Stuff() { Id = Guid.NewGuid(), SomeInt = new Random().Next(0, 10), SomeString = "yeah, a string" };
            for (int i = 0; i < stuff.SomeInt; i++)
                stuff.ListOfStrings.Add(i.ToString());
            return stuff;
        }

        private MongoCollection GetMongoCollection()
        {
            var client = new MongoClient("mongodb://localhost");
            var server = client.GetServer();
            var database = server.GetDatabase("BlogMongo");
            return database.GetCollection<Stuff>("stuffses");
        }

        private void SaveStuff()
        {
            var stuff = GenerateStuff();
            var collection = GetMongoCollection();
            collection.Save(stuff);
        }

        private void QueryStuff()
        {
            var collection = GetMongoCollection();
            var singleQuery = Query<Stuff>.EQ(e => e.SomeInt, 3);
            var stuff = collection.FindOneAs<Stuff>(singleQuery);
            var serializer = new JavaScriptSerializer();
            Response.Write(serializer.Serialize(stuff));
            var linqQuery = from item in collection.AsQueryable<Stuff>()
                            where item.SomeString == "yeah, a string"
                            select item;
            foreach (var item in linqQuery)
                Response.Write(serializer.Serialize(item));
        }

        protected void btnQuery_Click(object sender, EventArgs e)
        {
            QueryStuff();
        }

        protected void btnSave_Click(object sender, EventArgs e)
        {
            SaveStuff();
        }

    }
}


What's Next

There are plenty of other ways to query a MongoDB instance. Play around with it and see what you can find, and read the resource links below. The Mongo C# LINQ tutorial and the Mongo C# Driver Getting Started pages are both very handy. This is my last blog on NoSQL (at least for now) though, so we'll be on to a different topic next week. If you want further NoSQL knowledge, then you're on your own buddy!

Resources

MongoDB C# Driver LINQ Tutorial
MongoDB Getting Started With the C# Driver

Thursday, July 17, 2014

NoSQL - MongoDB; Setup and Saving Data

Intro

In the previous post in this series I covered the basics of NoSQL data stores. I briefly mentioned MongoDB and its home in the NoSQL world as a document store. This week I'll go over some of the basics of using MongoDB in your c# code. Hold onto yer hats folks, we're in for a ride!

Setting up Mongo Locally

Go to the MongoDB download page, download and setup the correct version of Mongo for your OS. The instructions on their site are better than anything I can write, so do what they say and you'll be fine :). Be sure to actually run the MongoDB system after you install it (there are instructions for running the program on their website too), as it's pretty tough to connect to a database that isn't running.

A Free MongoDB GUI

MongoDB does not come with a GUI. If you've got a SQL Server background you've probably become accustomed to visualizing your data using SQL Server Management Studio, as it's a very handy GUI for navigating around your data. Mongo does have a couple options, they just don't come with MongoDB. For this tutorial, I'll be using MongoVue, which I suggest you download too for your own usage. I won't get into general usage of this program as it's pretty easy and it's detailed nicely on their website.

A Quick Bit of Mongo Terminology

If you have a SQL background you're well on your way to understanding how a MongoDB server is organized. The only thing that may throw you off initially is Mongo doesn't use tables. The closes thing they have is a Collection, which is roughly akin to a table. You're supposed to store only a single type of object within each collection, though it's not enforced. Still, it's a good idea as it keeps your data organized.

Saving Data

OK then cool cats, it's time to fire up Visual Studio and play with Mongo. Create yourself a new webforms project. I called mine BlogMongo, but call yours whatever you like. Now go into NuGet package manager and install the package "mongocsharpdriver" into your project. This NuGet package contains everything you need to read and write data using MongoDB and C#. It's the official package listed on the MongoDB site, so it's my preferred option though there are others out there.

Drop a server-side button on your webform.Call the button btnSave. Add a click() event to it.Add a couple using statements too; one for MongoDB.Bson and one for MongoDB.Driver. Now I'll work a little coding magic, paste it in here, and discuss below:

using MongoDB.Bson;
using MongoDB.Driver;
using System;
using System.Collections.Generic;

namespace BlogMongo
{
    public class Stuff
    {
        public Guid Id { get; set; }
        public int SomeInt { get; set; }
        public string SomeString { get; set; }
        public IList<string> ListOfStrings { get; set; }
        public Stuff()
        {
            ListOfStrings = new List<string>();
        }
    }

    public partial class Default : System.Web.UI.Page
    {
        private Stuff GenerateStuff()
        {
            var stuff = new Stuff() { Id = Guid.NewGuid(), SomeInt = new Random().Next(0, 10), SomeString = "yeah, a string" };
            for (int i = 0; i < stuff.SomeInt; i++)
                stuff.ListOfStrings.Add(i.ToString());
            return stuff;
        }

        private MongoCollection GetMongoCollection()
        {
            var client = new MongoClient("mongodb://localhost");
            var server = client.GetServer();
            var database = server.GetDatabase("BlogMongo");
            return database.GetCollection<Stuff>("stuffses");
        }

        private void SaveStuff()
        {
            var stuff = GenerateStuff();
            var collection = GetMongoCollection();
            collection.Save(stuff);
        }

        protected void btnSave_Click(object sender, EventArgs e)
        {
            SaveStuff();
        }

        protected void Page_Load(object sender, EventArgs e)
        {
        }
    }
}



You'll see after the using statements, the first thing we've done is add a class called Stuff. This is just our POCO (plain old CLR object) that we'll store in the database. The GenerateStuff() method within our page merely creates an object of type Stuff and puts some data in it.

GetMongoCollection() is where things start to get interesting. The first line in the function gets us a reference to an MongoClient object, which is how you connect to a Mongo database. The code is connecting to the local installation we did up above. Line 2 of the method gets a reference to the Mongo server. In this case it's the same machine, but hey we need an object. Line 3 gets a reference to a database named BlogMongo within the Mongo server. Those of you reading carefully might be thinking "uh...Pete, forget something? Like maybe creating the database?". Nay friends, nay! I forgot no such thing! Call it a feature or call it a piece of kryptonite, but Mongo will create a database automatically the first time you save some data into it. So, even though this database doesn't yet exist, we can reference it. Neato! Line 4 retrieves from the database a collection (or table if you'd rather) named stuffses. Note that this collection doesn't exist yet either; the same rule for database creation applies to collection creation.

The next interesting method is SaveStuff(). It's pretty compact at 3 lines, and the 3rd line is the only really new bit of awesome. This uses our collection reference that we created above and calls its Save() method in order to save our stuff object which is of type Stuff. The Save() method is used for an upsert. There are separate Insert() and Update() methods, but I prefer to use this multipurpose method.

Now, just to prove the magic is still alive in my relationship with the code, I ran this little fella and clicked the button 4 times. Here is the representation of our data in MongoVUE:


A quick note on the storage behind MongoDB: MongoDB stores its data in BSON format, which is Binary JSON. It's just a flavor of JSON with a little bit of extra bells and whistles. That's why, in the above screenshot of MongoVue, I elected to show you the JSON representation of our stored objects. JSON is a pretty universal communication mechanism these days, so getting used to seeing it an using it can only benefit you.

Ok one last quick note, then I'm done noting. Really: See how there is no field named Id in my stored objects in that MongoVue screenshot, even though there is a field in the class Stuff named Id? This is because the Mongo C# driver will automatically use any field named Id as your primary key/id field, which is how documents are identified within the MongoDB database. MongoDB however stores such fields with a name of _id unless you specify otherwise, so in this case the field Id (which is a useful standard, so name your Id fields this if you can) maps to _id in the MongoDB database.

What's Next?

Next week will be the final installment of NoSQL/Mongo. I'll show you guys how to query data. Maybe a few other tricks too if I have the time and we have enough space on the blog post.

I encourage you to read up on using the Mongo C# driver using the link in the resources section below.We only scratched the surface this week, and we'll lightly gouge it next week. There's still plenty we won't have time for though, so if you like MongoDB and you want to learn more, their website is the best place to get into it.

Resources

MongoDB-Getting Started With the C# Driver

Thursday, July 10, 2014

NoSQL - A Brief Introduction

Intro

What is a NoSQL Database? It's generally considered to be a data storage system that doesn't use relational tables and isn't accessed by structured query language. For decades now SQL and relational tables have been the standard for data storage. SQL storage, or more generally speaking relational storage, is driven by the concept of denormalization of data to reduce redundancy. You have many tables representing a structure of objects/entities, and you can join those many tables together to bring back the entire picture of your data and objects.

Why has NoSQL Become Big News?

Some larger companies such as Facebook, Twitter, and many others have found that modeling their data using a relational database system just wasn't fast enough and didn't scale well with load, and it is quite complex to the casual observer. And, the part I've found most tiresome in my own work, a relational database enforces a rigid scheme. Have you ever spent much time mapping a huge object tree to the tables and columns behind it? It's not terribly fun.

The big relational database systems have spent a lot of time and money optimizing their systems, but at a certain point they become constrained by the disk system behind them. Once your disks are your bottleneck, the only options are to get more expensive and faster storage subsystems (ie faster drives or a faster/better SAN-type system). Plus, if you have a complicated object hierarchy with many levels and relations to model, you have more and more tables to model which means more joining of tables and more processing/disk hits to get your data. There are coding frameworks for dealing with the mapping side of things, such as Entity Framework, but the complexity of the mapping is still there; the tedium is just automated for you.

Types of NoSQL Databases

NoSQL systems have been created with the idea of tackling these problems in various ways. Now this doesn't mean that relational databases are obsolete; rather, with the growing maturity and popularity of various NoSQL platforms, you have more options available to research and choose from. As with many technologies, the more you learn the better decisions you can make so I encourage you to keep reading and see if you can think of some ideas of how you might use NoSQL in your own projects.

There are many different types of NoSQL databases, but I will only give a brief overview of three of them:

  1. Document Store: The general concept behind a document store is that objects/entities are represented in the underlying storage mechanism with a "document". Generally speaking you are encouraged to store your entire object (and all it's children, and all it's children, etc) in a single document. These documents are in various formats depending on the vendor. You might see JSON with one vendor, BSON with another, XML, etc. If you've ever exported an object to JSON, you can visualize a document store roughly as a collection of JSON exports (serialized objects) on a disk system, that you can query with some sort of vendor-proprietary query language. 
    1. Benefits: The perceived benefits of a document store over a relational store are, of course, dependent on how you use the system. Imagine if you have a fairly simple base storage object (a Chair) with 7 child objects (5 wheels, a single bolt, and a single arm; not a comfortable chair I know). Now mentally compare loading this from a relational database with loading it as a single JSON document. To load from a relational database you have to join together (or query separately) 8 different tables. This means at least 8 separate hits to your underlying disk storage system, probably more with indexing. To load the same object from a document store? The object and all of its children are stored in a single document, so you have 1 disk operation. Oh and did I mention that it almost eliminates object-->storage mapping problems? Because you are, for the most part, just serializing your object to some form of document and saving that whole thing, you don't really care what's in that document. It's all taken care of for you. No mapping!
    2. Pitfalls: These can of course vary based on the vendor, but in general you have to have a good idea up-front of what you are going to want to do with your data further down the road or you will end up with a lot of duplicated data that is hard to maintain. Let's say for example that within the arm sub-object of the chair you store the name of the arm's manufacturer. If you have 20 chairs in your database, that's ~20 references to the name of this manufacturer (assuming all 20 chairs use the same type of chair arm) in your storage system. What if the manufacturer changes its name? You now have to go and modify all 20 of the chairs/arms in your storage system. In a relational system with a normalized structure, you modify the one chair arm entry as it's just a single row in a table of chair arms. So, a general rule with document stores is they're best-used when your top-level object is what you care about. If you care about treating lower-level objects as primary citizens of your project ecosystem, you should consider other alternatives. (tip: you can in fact have multiple types of collections of objects in a document store and relate them to each other, but that's a topic for another day).
  2. Graph: The big idea behind a graph database is that it allows for a more efficient modeling of relationships between objects. Every element has a direct link to adjacent elements, and through some magic I don't quite understand this means you don't need to have foreign keys and indexes on them. Thus, lookups of relationships are quite fast. Where might this be useful? Social relationship mapping (I'm talkin bout you Facebook), transportation, etc. 
    1. Benefits: Clearly the benefit here is for representing entities/objects that are tightly related and need to be referenced together with their relations. The specific improvement here is speed. A side effect of the better performance is that they will also scale better (continue to have good performance) even as the data set grows much larger.
    2. Pitfalls: Relational databases are, in general, going to be better at performing summary/group calculations and updates of large amounts of data at the same time. Be sure you're using the right tool for what your system needs to accomplish.
  3. Key-Value Store: As the name implies, a key value store represents the stored entities as a list of key/value pairs. For those of you familiar with the .net world, think of the dictionary class. The key is some sort of value that uniquely identifies the data/value, and the value is whatever you want it to be. Serialized object, an int, whatever. A Key-Value store is almost like an even simpler version of a document store. With a document store the database doesn't care what the structure of your document is, but the document does have a structure and the database can (most of the time) query against it. With a key-value store, the database doesn't know or care what's in there and you can't query on it. You can only pull up a value by key, that's it. Nothing fancy.
    1. Benefits: Very simple and very flexible in terms of what you can store. Because there is no structure whatsoever to the data, you can put whatever you feel like in the value portion of a key-value store.
    2. Drawbacks: Inflexible in terms of how you retrieve data. You'd better be sure you don't need to write any ad-hoc queries on your data, because you can only load a value or not; that's it.


My Experience


I myself have a very limited exposure to NoSQL databases. In fact, I've never used one in production, and I've only spent a few hours developing with them on personal projects. So far my experience with NoSQL has been limited to two specific document store databases, Couchbase and MongoDB.

In general, I have to say I have found both of them quite easy to develop with. Querying data from them is relatively straightfoward, though if you're used to SQL you will have to learn new syntax and all that jazz; sorry. Setup is usually pretty easy, though due to a rather nasty bug at the time I was using Couchbase, MongoDB proved much easier to get up and running consistently in a Windows development environment.

The most useful piece of advice I can give regarding NoSQL databases is: read up about them through some google-fu, learn how they work, and research how others have used them with both good and bad results. Then see if and how you can apply them to what you do. They do have their uses, but be careful and ask for advice before implementing.


Resources


NoSQL
Document Oriented Database
Graph Database
Key-Value Store