Quantcast
Channel: Rick Strahl's Web Log
Viewing all 664 articles
Browse latest View live

Basic Spatial Data with SQL Server and Entity Framework 5.0

$
0
0

In my most recent project we needed to do a bit of geo-spatial referencing. While spatial features have been in SQL Server for a while using those features inside of .NET applications hasn't been as straight forward as could be, because .NET natively doesn't support spatial types. There are workarounds for this with a few custom project like SharpMap or a hack using the Sql Server specific Geo types found in the Microsoft.SqlTypes assembly that ships with SQL server.

While these approaches work for manipulating spatial data from .NET code, they didn't work with database access if you're using Entity Framework. Other ORM vendors have been rolling their own versions of spatial integration. In Entity Framework 5.0 running on .NET 4.5 the Microsoft ORM finally adds support for spatial types as well.

In this post I'll describe basic geography features that deal with single location and distance calculations which is probably the most common usage scenario.

SQL Server Transact-SQL Syntax for Spatial Data

Before we look at how things work with Entity framework, lets take a look at how SQL Server allows you to use spatial data to get an understanding of the underlying semantics. The following SQL examples should work with SQL 2008 and forward.

Let's start by creating a test table that includes a Geography field and also a pair of Long/Lat fields that demonstrate how you can work with the geography functions even if you don't have geography/geometry fields in the database. Here's the CREATE command:

CREATE TABLE [dbo].[Geo](
    [id] [int] IDENTITY(1,1) NOT NULL,
    [Location] [geography] NULL,
    [Long] [float] NOT NULL,
    [Lat] [float] NOT NULL
)

Now using plain SQL you can insert data into the table using geography::STGeoFromText SQL CLR function:

insert into Geo( Location , long, lat ) values
               ( geography::STGeomFromText ('POINT(-121.527200 45.712113)', 4326), -121.527200, 45.712113 )
insert into Geo( Location , long, lat ) values 
               ( geography::STGeomFromText ('POINT(-121.517265 45.714240)', 4326), -121.517265, 45.714240 )
insert into Geo( Location , long, lat ) values 
               ( geography::STGeomFromText ('POINT(-121.511536 45.714825)', 4326), -121.511536, 45.714825)


The STGeomFromText function accepts a string that points to a geometric item (a point here but can also be a line or path or polygon and many others). You also need to provide an SRID (Spatial Reference System Identifier) which is an integer value that determines the rules for how geography/geometry values are calculated and returned. For mapping/distance functionality you typically want to use 4326 as this is the format used by most mapping software and geo-location libraries like Google and Bing.

The spatial data in the Location field is stored in binary format which looks something like this:

spatialresults

Once the location data is in the database you can query the data and do simple distance computations very easily. For example to calculate the distance of each of the values in the database to another spatial point is very easy to calculate.

Distance calculations compare two points in space using a direct line calculation. For our example I'll compare a new point to all the points in the database.

Using the Location field the SQL looks like this:

-- create a source point
DECLARE @s geography
SET @s = geography:: STGeomFromText('POINT(-121.527200 45.712113)' , 4326); --- return the ids select ID, Location as Geo , Location .ToString() as Point , @s.STDistance( Location) as distance from Geo order by distance

The code defines a new point which is the base point to compare each of the values to. You can also compare values from the database directly, but typically you'll want to match a location to another location and determine the difference for which you can use the geography::STDistance function.

This query produces the following output:

DistanceQuery

The STDistance function returns the straight line distance between the passed in point and the point in the database field. The result for SRID 4326 is always in meters. Notice that the first value passed was the same point so the difference is 0. The other two points are two points here in town in Hood River a little ways away - 808 and 1256 meters respectively.

Notice also that you can order the result by the resulting distance, which effectively gives you results that are ordered radially out from closer to further away. This is great for searches of points of interest near a central location (YOU typically!).

These geolocation functions are also available to you if you don't use the Geography/Geometry types, but plain float values. It's a little more work, as each point has to be created in the query using the string syntax, but the following code doesn't use a geography field but produces the same result as the previous query.

--- using float fields
select ID,
    geography::STGeomFromText ('POINT(' + STR (long, 15,7 ) + ' ' + Str(lat ,15, 7) + ')' , 4326),
    geography::STGeomFromText ('POINT(' + STR (long, 15,7 ) + ' ' + Str(lat ,15, 7) + ')' , 4326). ToString(),        
    @s.STDistance( geography::STGeomFromText ('POINT(' + STR(long ,15, 7) + ' ' + Str(lat ,15, 7) + ')' , 4326)) as distance          
from geo
order by distance

Spatial Data in the Entity Framework

Prior to Entity Framework 5.0 on .NET 4.5 consuming of the data above required using stored procedures or raw SQL commands to access the spatial data. In Entity Framework 5 however, Microsoft introduced the new DbGeometry and DbGeography types. These immutable location types provide a bunch of functionality for manipulating spatial points using geometry functions which in turn can be used to do common spatial queries like I described in the SQL syntax above.

The DbGeography/DbGeometry types are immutable, meaning that you can't write to them once they've been created. They are a bit odd in that you need to use factory methods in order to instantiate them - they have no constructor() and you can't assign to properties like Latitude and Longitude.

Creating a Model with Spatial Data

Let's start by creating a simple Entity Framework model that includes a Location property of type DbGeography:

    public class GeoLocationContext : DbContext
    {
        public DbSet<GeoLocation> Locations { get; set; }
    }

    public class GeoLocation
    {
        public int Id { get; set; }
        public DbGeography Location { get; set; }
        public string Address { get; set; }
    }

That's all there's to it. When you run this now against SQL Server, you get a Geography field for the Location property, which looks the same as the Location field in the SQL examples earlier.

Adding Spatial Data to the Database

Next let's add some data to the table that includes some latitude and longitude data. An easy way to find lat/long locations is to use Google Maps to pinpoint your location, then right click and click on What's Here. Click on the green marker to get the GPS coordinates.

To add the actual geolocation data create an instance of the GeoLocation type and use the DbGeography.PointFromText() factory method to create a new point to assign to the Location property:

[TestMethod]
public void AddLocationsToDataBase()
{
    var context = new GeoLocationContext();

    // remove all
    context.Locations.ToList().ForEach( loc => context.Locations.Remove(loc));
    context.SaveChanges();

    var location = new GeoLocation()
    {
        // Create a point using native DbGeography Factory method
        Location = DbGeography.PointFromText(
                    string.Format("POINT({0} {1})", -121.527200,45.712113)
                    ,4326),
        Address = "301 15th Street, Hood River"
    };
    context.Locations.Add(location);

    location = new GeoLocation()
    {
        Location = CreatePoint(45.714240, -121.517265),
        Address = "The Hatchery, Bingen"
    };
    context.Locations.Add(location);

    location = new GeoLocation()
    {
        // Create a point using a helper function (lat/long)
        Location = CreatePoint(45.708457, -121.514432),
        Address = "Kaze Sushi, Hood River"
    };
    context.Locations.Add(location);

    location = new GeoLocation()
    {
        Location = CreatePoint(45.722780, -120.209227),
        Address = "Arlington, OR"
    };
    context.Locations.Add(location);

    context.SaveChanges();
}

As promised, a DbGeography object has to be created with one of the static factory methods provided on the type as the Location.Longitude and Location.Latitude properties are read only. Here I'm using PointFromText() which uses a "Well Known Text" format to specify spatial data. In the first example I'm specifying to create a Point from a longitude and latitude value, using an SRID of 4326 (just like earlier in the SQL examples).

You'll probably want to create a helper method to make the creation of Points easier to avoid that string format and instead just pass in a couple of double values. Here's my helper called CreatePoint that's used for all but the first point creation in the sample above:

public static DbGeography CreatePoint(double latitude, double longitude)
{
    var text = string.Format(CultureInfo.InvariantCulture.NumberFormat,
                             "POINT({0} {1})", longitude, latitude);
    // 4326 is most common coordinate system used by GPS/Maps
    return DbGeography.PointFromText(text, 4326);
}

Using the helper the syntax becomes a bit cleaner, requiring only a latitude and longitude respectively. Note that my method intentionally swaps the parameters around because Latitude and Longitude is the common format I've seen with mapping libraries (especially Google Mapping/Geolocation APIs with their LatLng type).

When the context is changed the data is written into the database using the SQL Geography type which looks the same as in the earlier SQL examples shown.

Querying

Once you have some location data in the database it's now super easy to query the data and find out the distance between locations. A common query is to ask for a number of locations that are near a fixed point - typically your current location and order it by distance.

Using LINQ to Entities a query like this is easy to construct:

[TestMethod]
public void QueryLocationsTest()
{
    var sourcePoint = CreatePoint(45.712113, -121.527200);

    var context = new GeoLocationContext();

    // find any locations within 5 kilometers ordered by distance
    var matches = 
        context.Locations
                .Where(loc => loc.Location.Distance(sourcePoint) < 5000)
                .OrderBy( loc=> loc.Location.Distance(sourcePoint) )
                .Select( loc=> new { Address = loc.Address, Distance = loc.Location.Distance(sourcePoint) });

    Assert.IsTrue(matches.Count() > 0);

    foreach (var location in matches)
    {
        Console.WriteLine("{0} ({1:n0} meters)", location.Address, location.Distance);
    }
}

This example produces:

301 15th Street, Hood River (0 meters)
The Hatchery, Bingen (809 meters)
Kaze Sushi, Hood River (1,074 meters)

 

The first point in the database is the same as my source point I'm comparing against so the distance is 0. The other two are within the 5 mile radius, while the Arlington location which is 65 miles or so out is not returned. The result is ordered by distance from closest to furthest away.

In the code, I first create a source point that is the basis for comparison. The LINQ query then selects all locations that are within 5km of the source point using the Location.Distance() function, which takes a source point as a parameter. You can either use a pre-defined value as I'm doing here, or compare against another database DbGeography property (say when you have to points in the same database for things like routes).

What's nice about this query syntax is that it's very clean and easy to read and understand. You can calculate the distance and also easily order by the distance to provide a result that shows locations from closest to furthest away which is a common scenario for any application that places a user in the context of several locations. It's now super easy to accomplish this.

Meters vs. Miles

As with the SQL Server functions, the Distance() method returns data in meters, so if you need to work with miles or feet you need to do some conversion. Here are a couple of helpers that might be useful (can be found in GeoUtils.cs of the sample project):

/// <summary>
/// Convert meters to miles
/// </summary>
/// <param name="meters"></param>
/// <returns></returns>
public static double MetersToMiles(double? meters)
{
    if (meters == null)
        return 0F;

    return meters.Value * 0.000621371192;
}

/// <summary>
/// Convert miles to meters
/// </summary>
/// <param name="miles"></param>
/// <returns></returns>
public static double MilesToMeters(double? miles)
{
    if (miles == null)
        return 0;

    return miles.Value * 1609.344;
}

Using these two helpers you can query on miles like this:

[TestMethod]
public void QueryLocationsMilesTest()
{
    var sourcePoint = CreatePoint(45.712113, -121.527200);

    var context = new GeoLocationContext();

    // find any locations within 5 miles ordered by distance
    var fiveMiles = GeoUtils.MilesToMeters(5);

    var matches =
        context.Locations
                .Where(loc => loc.Location.Distance(sourcePoint) <= fiveMiles)
                .OrderBy(loc => loc.Location.Distance(sourcePoint))
                .Select(loc => new { Address = loc.Address, Distance = loc.Location.Distance(sourcePoint) });

    Assert.IsTrue(matches.Count() > 0);

    foreach (var location in matches)
    {
        Console.WriteLine("{0} ({1:n1} miles)", location.Address, GeoUtils.MetersToMiles(location.Distance));
    }
}

which produces:

301 15th Street, Hood River (0.0 miles)
The Hatchery, Bingen (0.5 miles)
Kaze Sushi, Hood River (0.7 miles)

Nice 'n simple.

.NET 4.5 Only

Note that DbGeography and DbGeometry are exclusive to Entity Framework 5.0 (not 4.4 which ships in the same NuGet package or installer) and requires .NET 4.5. That's because the new DbGeometry and DbGeography (and related) types are defined in the 4.5 version of System.Data.Entity which is a CLR assembly and is only updated by major versions of .NET. Why this decision was made to add these types to System.Data.Entity rather than to the frequently updated EntityFramework assembly that would have possibly made this work in .NET 4.0 is beyond me, especially given that there are no native .NET framework spatial types to begin with.

I find it also odd that there is no native CLR spatial type. The DbGeography and DbGeometry types are specific to Entity Framework and live on those assemblies. They will also work for general purpose, non-database spatial data manipulation, but then you are forced into having a dependency on System.Data.Entity, which seems a bit silly. There's also a System.Spatial assembly that's apparently part of WCF Data Services which in turn don't work with Entity framework. Another example of multiple teams at Microsoft not communicating and implementing the same functionality (differently) in several different places. Perplexed as a I may be, for EF specific code the Entity framework specific types are easy to use and work well.

Working with pre-.NET 4.5 Entity Framework and Spatial Data

If you can't go to .NET 4.5 just yet you can also still use spatial features in Entity Framework, but it's a lot more work as you can't use the DbContext directly to manipulate the location data. You can still run raw SQL statements to write data into the database and retrieve results using the same TSQL syntax I showed earlier using Context.Database.ExecuteSqlCommand().

Here's code that you can use to add location data into the database:

[TestMethod]
public void RawSqlEfAddTest()
{
    string sqlFormat = 
    @"insert into GeoLocations( Location, Address) values
            ( geography::STGeomFromText('POINT({0} {1})', 4326),@p0 )";

    var sql = string.Format(sqlFormat,-121.527200, 45.712113);

    Console.WriteLine(sql);

    var context = new GeoLocationContext();

    Assert.IsTrue(context.Database.ExecuteSqlCommand(sql,"301 N. 15th Street") > 0);
}

Here I'm using the STGeomFromText() function to add the location data.

Note that I'm using string.Format here, which usually would be a bad practice but is required here. I was unable to use ExecuteSqlCommand() and its named parameter syntax as the longitude and latitude parameters are embedded into a string. Rest assured it's required as the following does not work:

string sqlFormat = 
    @"insert into GeoLocations( Location, Address) values
            ( geography::STGeomFromText('POINT(@p0 @p1)', 4326),@p2 )";
context.Database.ExecuteSqlCommand(sql, -121.527200, 45.712113, "301 N. 15th Street")

Explicitly assigning the point value with string.format works however.

There are a number of ways to query location data. You can't get the location data directly, but you can retrieve the point string (which can then be parsed to get Latitude and Longitude) and you can return calculated values like distance.

Here's an example of how to retrieve some geo data into a resultset using EF's and SqlQuery method:

[TestMethod]
public void RawSqlEfQueryTest()
{
    var sqlFormat =
    @"
    DECLARE @s geography
    SET @s = geography:: STGeomFromText('POINT({0} {1})' , 4326);

    SELECT     
        Address,        
        Location.ToString() as GeoString,        
        @s.STDistance( Location) as Distance        
    FROM GeoLocations
    ORDER BY Distance";

    var sql = string.Format(sqlFormat, -121.527200, 45.712113);    

    var context = new GeoLocationContext();
    var locations = context.Database.SqlQuery<ResultData>(sql);

    Assert.IsTrue(locations.Count() > 0);

    foreach (var location in locations)
    {
        Console.WriteLine(location.Address + " " + location.GeoString + " " + location.Distance);
    }
}

public class ResultData
{
    public string GeoString { get; set; }
    public double Distance { get; set; }
    public string Address { get; set; }
}

Hopefully you don't have to resort to this approach as it's fairly limited. Using the new DbGeography/DbGeometry types makes this sort of thing so much easier. When I had to use code like this before I typically ended up retrieving data pks only and then running another query with just the PKs to retrieve the actual underlying DbContext entities. This was very inefficient and tedious but it did work.

Summary

For the current project I'm working on we actually made the switch to .NET 4.5 purely for the spatial features in EF 5.0. This app heavily relies on spatial queries and it was worth taking a chance with pre-release code to get this ease of integration as opposed to manually falling back to stored procedures or raw SQL string queries to return spatial specific queries. Using native Entity Framework code makes life a lot easier than the alternatives. It might be a late addition to Entity Framework, but it sure makes location calculations and storage easy. Where do you want to go today? ;-)

Resources

© Rick Strahl, West Wind Technologies, 2005-2012
Posted in ADO.NET  Sql Server  .NET  

.NET HTML Sanitation for rich HTML Input

$
0
0

Recently I was working on updating a legacy application to MVC 4 that included free form text input. When I set up the new site my initial approach was to not allow any rich HTML input, only simple text formatting that would respect a few simple HTML commands for bold, lists etc. and automatically handles line break processing for new lines and paragraphs. This is typical for what I do with most multi-line text input in my apps and it works very well with very little development effort involved.

Then the client sprung another note: Oh by the way we have a bunch of customers (real estate agents) who need to post complete HTML documents. Oh uh! There goes the simple theory. After some discussion and pleading on my part (<snicker>) to try and avoid this type of raw HTML input because of potential XSS issues, the client decided to go ahead and allow raw HTML input anyway.

There has been lots of discussions on this subject on StackOverFlow (and here and here) but to after reading through some of the solutions I didn't really find anything that would work even closely for what I needed. Specifically we need to be able to allow just about any HTML markup, with the exception of script code. Remote CSS and Images need to be loaded, links need to work and so. While the 'legit' HTML posted by these agents is basic in nature it does span most of the full gamut of HTML (4). Most of the solutions XSS prevention/sanitizer solutions I found were way to aggressive and rendered the posted output unusable mostly because they tend to strip any externally loaded content.

In short I needed a custom solution. I thought the best solution to this would be to use an HTML parser - in this case the Html Agility Pack - and then to run through all the HTML markup provided and remove any of the blacklisted tags and a number of attributes that are prone to JavaScript injection.

There's much discussion on whether to use blacklists vs. whitelists in the discussions mentioned above, but I found that whitelists can make sense in simple scenerios where you might allow HTML input, but when you need to allow a larger array of HTML functionality a blacklist is probably an easier and more solution. Also white listing gets a bit more complex with HTML5 and the new proliferation of new HTML tags.

The Microsoft Web Protection Library (AntiXSS)

My first thought was to check out the Microsoft AntiXSS library. Microsoft has an HTML Encoding and Sanitation library in the Microsoft Web Protection Library (formerly AntiXSS Library) on CodePlex, which provides stricter functions for whitelist encoding and sanitation. Initially I thought the Sanitation class and its static members would do the trick for me,but I found that this library is way too restrictive for my needs. Specifically the Sanitation class strips out images and links which rendered the full HTML from our real estate clients completely useless. I didn't spend much time with it, but apparently I'm not alone if feeling this library is not really useful without some way to configure operation.

To give you an example of what didn't work for me with the library here's a HTML fragment that includes script, img and a tags where I would expect the script to be stripped and leave everything else intact:

var value = "<b>Here</b> <script>alert('hello')</script> we go. Visit the " + 
"<a href='http://west-wind.com'>West Wind</a> site. " +
            "<img src='http://west-wind.com/images/new.gif' /> " ;

and then using:

@Html.Raw(Microsoft.Security.Application.Sanitizer.GetSafeHtmlFragment(value))

produced a not so useful sanitized string:

Here we go. Visit the <a>West Wind</a> site.

While it removed the <script> tag (good) it also removed the href from the link and the image tag altogether (bad). In some situations this might be useful, but for most tasks I doubt this is the desired behavior. Links can be javascript and images can 'broadcast' information to a server, but without configuration to tell the library what to restrict this becomes useless to me. I couldn't find anyway to customize the white list, nor is there code available in this 'open source' library on CodePlex.

Using Html Agility Pack for HTML Parsing

So a custom solution it was! After doing a bit of research I decided the best approach for a custom solution would be to use an HTML parser and inspect the HTML fragment/document I'm trying to import. I've used the HTML Agility Pack before for a number of apps where I needed an HTML parser without requiring an instance of a full browser like the Internet Explorer Application object which is inadequate in Web apps. In case you haven't checked out the Html Agility Pack before, it's a powerful HTML parser library that you can use from your .NET code. It provides a simple, parseable HTML DOM model to full HTML documents or HTML fragments that let you walk through each of the elements in your document. If you've used the HTML or XML DOM in a browser before you'll feel right at home with the Agility Pack.

Blacklist based HTML Parsing to strip XSS Code

For my purposes of HTML sanitation, the process involved is to walk the HTML document one element at a time and then check each element and attribute against a blacklist. There's quite a bit of argument of what's better: A whitelist of allowed items or a blacklist of denied items. While whitelists tend to be more secure, they also require a lot more configuration. In the case of HTML5 a whitelist could be very extensive. For what I need, I only want to ensure that no JavaScript is executed, so a blacklist includes the obvious <script> tag plus any tag that allows loading of external content including <iframe>, <object>, <embed> and <link> etc. <form>  is also excluded to avoid posting content to a different location. I also disallow <head> and <meta> tags in particular for my case, since I'm only allowing posting of HTML fragments. There is also some internal logic to exclude some attributes or attributes that include references to javascript or CSS expressions.

The default tag blacklist reflects my use case, but is customizable and can be added to.

Here's my HtmlSanitizer implementation:

using System.Collections.Generic;
using System.IO;
using System.Xml;
using HtmlAgilityPack;

namespace Westwind.Web.Utilities
{
    public class HtmlSanitizer
    {

        public HashSet<string> BlackList = new HashSet<string>() 
        {
                { "script" },
                { "iframe" },
                { "form" },
                { "object" },
                { "embed" },
                { "link" },                
                { "head" },
                { "meta" }
        };

        /// <summary>
        /// Cleans up an HTML string and removes HTML tags in blacklist
        /// </summary>
        /// <param name="html"></param>
        /// <returns></returns>
        public static string SanitizeHtml(string html, params string[] blackList)
        {
            var sanitizer = new HtmlSanitizer();
            if (blackList != null && blackList.Length > 0)
            {
                sanitizer.BlackList.Clear();
                foreach (string item in blackList)
                    sanitizer.BlackList.Add(item);
            }
            return sanitizer.Sanitize(html);
        }

        /// <summary>
        /// Cleans up an HTML string by removing elements
        /// on the blacklist and all elements that start
        /// with onXXX .
        /// </summary>
        /// <param name="html"></param>
        /// <returns></returns>
        public string Sanitize(string html)
        {
            var doc = new HtmlDocument();
            
            doc.LoadHtml(html);            
            SanitizeHtmlNode(doc.DocumentNode);            
            
            //return doc.DocumentNode.WriteTo();

            string output = null;

            // Use an XmlTextWriter to create self-closing tags
            using (StringWriter sw = new StringWriter())
            {
                XmlWriter writer = new XmlTextWriter(sw);
                doc.DocumentNode.WriteTo(writer);
                output = sw.ToString();

                // strip off XML doc header
                if (!string.IsNullOrEmpty(output))
                {
                    int at = output.IndexOf("?>");
                    output = output.Substring(at + 2);
                }

                writer.Close();
            }
            doc = null;

            return output;
        }

        private void SanitizeHtmlNode(HtmlNode node)
        {
            if (node.NodeType == HtmlNodeType.Element)
            {
                // check for blacklist items and remove
                if (BlackList.Contains(node.Name))
                {
                    node.Remove();
                    return;
                }

                // remove CSS Expressions and embedded script links
                if (node.Name == "style")
                {
                    if (string.IsNullOrEmpty(node.InnerText))
                    {
                        if (node.InnerHtml.Contains("expression") || node.InnerHtml.Contains("javascript:"))
                            node.ParentNode.RemoveChild(node);
                    }
                }

                // remove script attributes
                if (node.HasAttributes)
                {
                    for (int i = node.Attributes.Count - 1; i >= 0; i--)
                    {
                        HtmlAttribute currentAttribute = node.Attributes[i];
                 
                        var attr = currentAttribute.Name.ToLower();
                        var val = currentAttribute.Value.ToLower();
                        
                        // remove event handlers
                        if (attr.StartsWith("on"))
                            node.Attributes.Remove(currentAttribute);
                        
                        // remove script links
                        else if (
                                 //(attr == "href" || attr== "src" || attr == "dynsrc" || attr == "lowsrc") &&
                                 val != null &&
                                 val.Contains("javascript:"))
                            node.Attributes.Remove(currentAttribute);
                        
                        // Remove CSS Expressions
                        else if (attr == "style" && 
                                 val != null &&
                                 val.Contains("expression") || val.Contains("javascript:") || val.Contains("vbscript:"))
                            node.Attributes.Remove(currentAttribute);
                    }
                }
            }

            // Look through child nodes recursively
            if (node.HasChildNodes)
            {
                for (int i = node.ChildNodes.Count - 1; i >= 0; i--)
                {
                    SanitizeHtmlNode(node.ChildNodes[i]);
                }
            }
        }
    }
}

Please note: Use this as a starting point only for your own parsing and review the code for your specific use case! If your needs are less lenient than mine were you can you can make this much stricter by not allowing src and href attributes or CSS links if your HTML doesn't allow it. You can also check links for external URLs and disallow those - lots of options.  The code is simple enough to make it easy to extend to fit your use cases more specifically.

The Sanitize method walks through each node of the document and then recursively drills into all of its children until the entire document has been traversed. Note that the code here uses an XmlTextWriter to write output - this is done to preserve XHTML style self-closing tags which are otherwise left as non-self-closing tags.

The sanitizer code scans for blacklist elements and removes those elements not allowed. Note that the blacklist is configurable either in the instance class as a property or in the static method via the parameter list. Additionally the code then goes through each element's attributes and looks for a host of rules gleaned from some of the XSS cheat sheets listed at the end of the post. Clearly there are a lot more XSS vulnerabilities, but a lot of them apply to ancient browsers (IE6 and versions of Netscape) - many of these glaring holes (like CSS expressions - WTF IE?) have been removed in recent browsers.

What a Pain

To be honest this is NOT something that I wanted to be doing. I think building anything related to XSS is better left to people who have far more knowledge of the topic than I do. Unfortunately, I was unable to find a tool that worked for me, or even provided a working base. For the project I was working on I had no choice and I'm sharing the code here merely as a base line to start with and potentially expand on for specific needs. It's sad that Microsoft Web Protection Library is currently such a train wreck - this is really something that should come from Microsoft as the systems vendor or possibly a third party that provides security tools.

Luckily for my application we are dealing with a authenticated and validated users so the user base is fairly well known, and relatively small - this is not a wide open Internet application that's directly public facing. As I mentioned earlier in the post, if I had my way I would simply not allow this type of raw HTML input in the first place, and instead rely on a more controlled HTML input mechanism like MarkDown or even a good HTML Edit control that can provide some limits on what types of input are allowed. Alas in this case I was overridden and we had to go forward with *any* raw HTML posted.

Sometimes I really feel sad that it's come this far - how many good applications and tools have been thwarted by fear of XSS attacks? So  many things that could be done *if* we had a more secure browser experience and didn't have to deal with every little script twerp trying to hack into Web pages. So much time wasted building secure apps, so much time wasted by others trying to hack apps… We're a funny species - no other species manages to waste as much time, effort and resources as we humans do :-)

Resources

© Rick Strahl, West Wind Technologies, 2005-2012
Posted in Security  HTML  ASP.NET  JavaScript  

Where does ASP.NET Web API Fit?

$
0
0

With the pending release of ASP.NET MVC 4 and the new ASP.NET Web API, there has been a lot of discussion of where the new Web API technology fits in the ASP.NET Web stack. There are a lot of choices to build HTTP based applications available now on the stack - we've come a long way from when WebForms and Http Handlers/Modules where the only real options. Today we have WebForms, MVC, ASP.NET Web Pages, ASP.NET AJAX, WCF REST and now Web API as well as the core ASP.NET runtime to choose to build HTTP content with.

Web API definitely squarely addresses the 'API' aspect - building consumable services - rather than HTML content, but even to that end there are a lot of choices you have today. So where does Web API fit, and when doesn't it?

But before we get into that discussion, let's talk about what a Web API is and why we should care.

What's a Web API?

HTTP 'APIs' (Microsoft's new terminology for a service I guess)  are becoming increasingly more important with the rise of the many devices in use today. Most mobile devices like phones and tablets run Apps that are using data retrieved from the Web over HTTP. Desktop applications are also moving in this direction with more and more online content and synching moving into even traditional desktop applications. The pending Windows 8 release promises an app like platform for both the desktop and other devices, that also emphasizes consuming data from the Cloud.

Likewise many Web browser hosted applications these days are relying on rich client functionality to create and manipulate the browser user interface, using AJAX rather than server generated HTML data to load up the user interface with data.

These mobile or rich Web applications use their HTTP connection to return data rather than HTML markup in the form of JSON or XML typically. But an API can also serve other kinds of data, like images or other binary files, or even text data and HTML (although that's less common). A Web API is what feeds rich applications with data.

ASP.NET Web API aims to service this particular segment of Web development by providing easy semantics to route and handle incoming requests and an easy to use platform to serve HTTP data in just about any content format you choose to create and serve from the server.

But .NET already has various HTTP Platforms

The .NET stack already includes a number of technologies that provide the ability to create HTTP service back ends, and it has done so since the very beginnings of the .NET platform. From raw HTTP Handlers and Modules in the core ASP.NET runtime, to high level platforms like ASP.NET MVC, Web Forms, ASP.NET AJAX and the WCF REST engine (which technically is not ASP.NET, but can integrate with it), you've always been able to handle just about any kind of HTTP request and response with ASP.NET. The beauty of the raw ASP.NET platform is that it provides you everything you need to build just about any type of HTTP application you can dream up from low level APIs/custom engines to high level HTML generation engine. ASP.NET as a core platform clearly has stood the test of time 10+ years later and all other frameworks like Web API are built on top of this ASP.NET core.

However, although it's possible to create Web APIs / Services using any of the existing out of box .NET technologies, none of them have been a really nice fit for building arbitrary HTTP based APIs. Sure, you can use an HttpHandler to create just about anything, but you have to build a lot of plumbing to build something more complex like a comprehensive API that serves a variety of requests, handles multiple output formats and can easily pass data up to the server in a variety of ways. Likewise you can use ASP.NET MVC to handle routing and creating content in various formats fairly easily, but it doesn't provide a great way to automatically negotiate content types and serve various content formats directly (it's possible to do with some plumbing code of your own but not built in). Prior to Web API, Microsoft's main push for HTTP services has been WCF REST, which was always an awkward technology that had a severe personality conflict, not being clear on whether it wanted to be part of WCF or purely a separate technology. In the end it didn't do either WCF compatibility or WCF agnostic pure HTTP operation very well, which made for a very developer-unfriendly environment.

Personally I didn't like any of the implementations at the time, so much so that I ended up building my own HTTP service engine (as part of the West Wind Web Toolkit), as have a few other third party tools that provided much better integration and ease of use. With the release of Web API for the first time I feel that I can finally use the tools in the box and not have to worry about creating and maintaining my own toolkit as Web API addresses just about all the features I implemented on my own and much more.

ASP.NET Web API provides a better HTTP Experience

ASP.NET Web API differentiates itself from the previous Microsoft in-box HTTP service solutions in that it was built from the ground up around the HTTP protocol and its messaging semantics. Unlike WCF REST or ASP.NET AJAX with ASMX, it’s a brand new platform rather than bolted on technology that is supposed to work in the context of an existing framework. The strength of the new ASP.NET Web API is that it combines the best features of the platforms that came before it, to provide a comprehensive and very usable HTTP platform. Because it's based on ASP.NET and borrows a lot of concepts from ASP.NET MVC, Web API should be immediately familiar and comfortable to most ASP.NET developers.

Here are some of the features that Web API provides that I like:

  • Strong Support for URL Routing to produce clean URLs using familiar MVC style routing semantics
  • Content Negotiation based on Accept headers for request and response serialization
  • Support for a host of supported output formats including JSON, XML, ATOM
  • Strong default support for REST semantics but they are optional
  • Easily extensible Formatter support to add new input/output types
  • Deep support for more advanced HTTP features via HttpResponseMessage and HttpRequestMessage
    classes and strongly typed Enums to describe many HTTP operations
  • Convention based design that drives you into doing the right thing for HTTP Services
  • Very extensible, based on MVC like extensibility model of Formatters and Filters
  • Self-hostable in non-Web applications 
  • Testable using testing concepts similar to MVC

Web API is meant to handle any kind of HTTP input and produce output and status codes using the full spectrum of HTTP functionality available in a straight forward and flexible manner.

Looking at the list above you can see that a lot of functionality is very similar to ASP.NET MVC, so many ASP.NET developers should feel quite comfortable with the concepts of Web API. The Routing and core infrastructure of Web API are very similar to how MVC works providing many of the benefits of MVC, but with focus on HTTP access and manipulation in Controller methods rather than HTML generation in MVC.

There’s much improved support for content negotiation based on HTTP Accept headers with the framework capable of detecting automatically what content the client is sending and requesting and serving the appropriate data format in return. This seems like such a little and obvious thing, but it's really important. Today's service backends often are used by multiple clients/applications and being able to choose the right data format for what fits best for the client is very important.

While previous solutions were able to accomplish this using a variety of mixed features of WCF and ASP.NET, Web API combines all this functionality into a single robust server side HTTP framework that intrinsically understands the HTTP semantics and subtly drives you in the right direction for most operations. And when you need to customize or do something that is not built in, there are lots of hooks and overrides for most behaviors, and even many low level hook points that allow you to plug in custom functionality with relatively little effort.

No Brainers for Web API

There are a few scenarios that are a slam dunk for Web API. If your primary focus of an application or even a part of an application is some sort of API then Web API makes great sense.

HTTP Services
If you're building a comprehensive HTTP API that is to be consumed over the Web, Web API is a perfect fit. You can isolate the logic in Web API and build your application as a service breaking out the logic into controllers as needed. Because the primary interface is the service there's no confusion of what should go where (MVC or API). Perfect fit.

Primary AJAX Backends
If you're building rich client Web applications that are relying heavily on AJAX callbacks to serve its data, Web API is also a slam dunk. Again because much if not most of the business logic will probably end up in your Web API service logic, there's no confusion over where logic should go and there's no duplication. In Single Page Applications (SPA), typically there's very little HTML based logic served other than bringing up a shell UI and then filling the data from the server with AJAX which means the business logic required for data retrieval and data acceptance and validation too lives in the Web API. Perfect fit.

Generic HTTP Endpoints
Another good fit are generic HTTP endpoints that to serve data or handle 'utility' type functionality in typical Web applications. If you need to implement an image server, or an upload handler in the past I'd implement that as an HTTP handler. With Web API you now have a well defined place where you can implement these types of generic 'services' in a location that can easily add endpoints (via Controller methods) or separated out as more full featured APIs. Granted this could be done with MVC as well, but Web API seems a clearer and more well defined place to store generic application services. This is one thing I used to do a lot of in my own libraries and Web API addresses this nicely. Great fit.

Mixed HTML and AJAX Applications: Not a clear Choice 

For all the commonality that Web API and MVC share they are fundamentally different platforms that are independent of each other. A lot of people have asked when does it make sense to use MVC vs. Web API when you're dealing with typical Web application that creates HTML and also uses AJAX functionality for rich functionality. While it's easy to say that all 'service'/AJAX logic should go into a Web API and all HTML related generation into MVC, that can often result in a lot of code duplication. Also MVC supports JSON and XML result data fairly easily as well so there's some confusion where that 'trigger point' is of when you should switch to Web API vs. just implementing functionality as part of MVC controllers.

Ultimately there's a tradeoff between isolation of functionality and duplication. A good rule of thumb I think works is that if a large chunk of the application's functionality serves data Web API is a good choice, but if you have a couple of small AJAX requests to serve data to a grid or autocomplete box it'd be overkill to separate out that logic into a separate Web API controller. Web API does add overhead to your application (it's yet another framework that sits on top of core ASP.NET) so it should be worth it

.Keep in mind that MVC can generate HTML and JSON/XML and just about any other content easily and that functionality is not going away, so just because you Web API is there it doesn't mean you have to use it. Web API is not a full replacement for MVC obviously either since there's not the same level of support to feed HTML from Web API controllers (although you can host a RazorEngine easily enough if you really want to go that route) so if you're HTML is part of your API or application in general MVC is still a better choice either alone or in combination with Web API.

I suspect (and hope) that in the future Web API's functionality will merge even closer with MVC so that you might even be able to mix functionality of both into single Controllers so that you don't have to make any trade offs, but at the moment that's not the case.

Some Issues To think about

Web API is similar to MVC but not the Same

Although Web API looks a lot like MVC it's not the same and some common functionality of MVC behaves differently in Web API. For example, the way single POST variables are handled is different than MVC and doesn't lend itself particularly well to some AJAX scenarios with POST data.

Code Duplication

I already touched on this in the Mixed HTML and Web API section, but if you build an MVC application that also exposes a Web API it's quite likely that you end up duplicating a bunch of code and - potentially - infrastructure. You may have to create authentication logic both for an HTML application and for the Web API which might need something different altogether. More often than not though the same logic is used, and there's no easy way to share. If you implement an MVC ActionFilter and you want that same functionality in your Web API you'll end up creating the filter twice.

AJAX Data or AJAX HTML

On a recent post's comments, David made some really good points regarding the commonality of MVC and Web API's and its place. One comment that caught my eye was a little more generic, regarding data services vs. HTML services. David says:

I see a lot of merit in the combination of Knockout.js, client side templates and view models, calling Web API for a responsive UI, but sometimes late at night that still leaves me wondering why I would no longer be using some of the nice tooling and features that have evolved in MVC ;-)

You know what - I can totally relate to that. On the last Web based mobile app I worked on, we decided to serve HTML partials to the client via AJAX for many (but not all!) things, rather than sending down raw data to inject into the DOM on the client via templating or direct manipulation. While there are definitely more bytes on the wire, with this, the overhead ended up being actually fairly small if you keep the 'data' requests small and atomic. Performance was often made up by the lack of client side rendering of HTML.

Server rendered HTML for AJAX templating gives so much better infrastructure support without having to screw around with 20 mismatched client libraries. Especially with MVC and partials it's pretty easy to break out your HTML logic into very small, atomic chunks, so it's actually easy to create small rendering islands that can be used via composition on the server, or via AJAX calls to small, tight partials that return HTML to the client. Although this is often frowned upon as to 'heavy', it worked really well in terms of developer effort as well as providing surprisingly good performance on devices. There's still plenty of jQuery and AJAX logic happening on the client but it's more manageable in small doses rather than trying to do the entire UI composition with JavaScript and/or 'not-quite-there-yet' template engines that are very difficult to debug. This is not an issue directly related to Web API of course, but something to think about especially for AJAX or SPA style applications.

Summary

Web API is a great new addition to the ASP.NET platform and it addresses a serious need for consolidation of a lot of half-baked HTTP service API technologies that came before it. Web API feels 'right', and hits the right combination of usability and flexibility at least for me and it's a good fit for true API scenarios. However, just because a new platform is available it doesn't meant that other tools or tech that came before it should be discarded or even upgraded to the new platform. There's nothing wrong with continuing to use MVC controller methods to handle API tasks if that's what your app is running now - there's very little to be gained by upgrading to Web API just because. But going forward Web API clearly is the way to go, when building HTTP data interfaces and it's good to see that Microsoft got this one right - it was sorely needed!

Resources

© Rick Strahl, West Wind Technologies, 2005-2012
Posted in Web Api  

Mapping UrlEncoded POST Values in ASP.NET Web API

$
0
0

If there's one thing that's a bit unexpected in ASP.NET Web API, it's the limited support for mapping url encoded POST data values to simple parameters of ApiController methods. When I first looked at this I thought I was doing something wrong, because it seems mighty odd that you can bind query string values to parameters by name, but can't bind POST values to parameters in the same way.

To demonstrate here's a simple example. If you have a Web API method like this:

[HttpGet]
public HttpResponseMessage Authenticate(string username, string password)
{
   …
}

and then hit with a URL like this:

http://localhost:88/samples/authenticate?Username=ricks&Password=sekrit

it works just fine. The query string values are mapped to the username and password parameters of our API method.

But if you now change the method to work with [HttpPost] instead like this:

[HttpPost]
public HttpResponseMessage Authenticate(string username, string password)
{
    …
}

and hit it with a POST HTTP Request like this:

POST http://localhost:88/samples/authenticate HTTP/1.1
Host: localhost:88
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Content-type: application/x-www-form-urlencoded
Content-Length: 30

Username=ricks&Password=sekrit

you'll find that while the request works, it doesn't actually receive the two string parameters. The username and password parameters are null and so the method is definitely going to fail.

When I mentioned this over Twitter a few days ago I got a lot of responses back of why I'd want to do this in the first place - after all HTML Form submissions are the domain of MVC and not WebAPI which is a valid point.

However, the more common use case is using POST Variables with AJAX calls. The following is quite common for passing simple values:

$.post(url,{ Username: "Rick", Password: "sekrit" },function(result) {…});

but alas that doesn't work.

How ASP.NET Web API handles Content Bodies

Web API supports parsing content data in a variety of ways, but it does not deal with multiple posted content values. In effect you can only post a single content value to a Web API Action method. That one parameter can be very complex and you can bind it in a variety of ways, but ultimately you're tied to a single POST content value in your parameter definition. While it's possible to support multiple parameters on a POST/PUT operation, only one parameter can be mapped to the actual content - the rest have to be mapped to route values or the query string.

Web API treats the whole request body as one big chunk of data that is sent to a Media Type Formatter that's responsible for de-serializing the content into whatever value the method requires. The restriction comes from async nature of Web API where the request data is read only once inside of the formatter that retrieves and deserializes it. Because it's read once, checking for content (like individual POST variables) first is not possible.

However, Web API does provide a couple of ways to access the form POST data:

  • Model Binding - object property mapping to bind POST values
  • FormDataCollection - collection of POST keys/values

ModelBinding POST Values - Binding POST data to Object Properties

The recommended way to handle POST values in Web API is to use Model Binding, which maps individual urlencoded POST values to properties of a model object provided as the parameter. Model binding requires a single object as input to be bound to the POST data, with each POST key that matches a property name (including nested properties like Address.Street) being mapped and updated including automatic type conversion of simple types. This is a very nice feature - and a familiar one from MVC - that makes it very easy to have model objects mapped directly from inbound data.

The obvious drawback with Model Binding is that you need a model for it to work: You have to provide a strongly typed object that can receive the data and this object has to map the inbound data.

To rewrite the example above to use ModelBinding I have to create a class maps the properties that I need as parameters:

public class LoginData
{
    public string Username { get; set; }
    public string Password { get; set; }
}

and then accept the data like this in the API method:

[HttpPost]
public HttpResponseMessage Authenticate(LoginData login)
{
    string username = login.Username;
    string password = login.Password;
    …
}

This works fine mapping the POST values to the properties of the login object.

As a side benefit of this method definition, the method now also allows posting of JSON or XML to the same endpoint. If I change my request to send JSON like this:

POST http://localhost:88/samples/authenticate HTTP/1.1
Host: localhost:88
Accept: application/json
Content-type: application/json Content-Length: 40 {"Username":"ricks","Password":"sekrit"}

it works as well and transparently, courtesy of the nice Content Negotiation features of Web API.

There's nothing wrong with using Model binding and in fact it's a common practice to use (view) model object for inputs coming back from the client and mapping them into these models.

But it can be  kind of a hassle if you have AJAX applications with a ton of backend hits, especially if many methods are very atomic and focused and don't effectively require a model or view. Not always do you have to pass structured data, but sometimes there are just a couple of simple response values that need to be sent back. If all you need is to pass a couple operational parameters, creating a view model object just for parameter purposes seems like overkill. Maybe you can use the query string instead (if that makes sense), but if you can't then you can often end up with a plethora of 'message objects' that serve no further  purpose than to make Model Binding work.

Note that you can accept multiple parameters with ModelBinding so the following would still work:

[HttpPost]
public HttpResponseMessage Authenticate(LoginData login, string loginDomain)

but only the object will be bound to POST data. As long as loginDomain comes from the querystring or route data this will work.

Collecting POST values with FormDataCollection

Another more dynamic approach to handle POST values is to collect POST data into a FormDataCollection. FormDataCollection is a very basic key/value collection (like FormCollection in MVC and Request.Form in ASP.NET in general) and then read the values out individually by querying each.

[HttpPost]
public HttpResponseMessage Authenticate(FormDataCollection form)
{
    var username = form.Get("Username");
    var password = form.Get("Password");
    …
}

The downside to this approach is that it's not strongly typed, you have to handle type conversions on non-string parameters, and it gets a bit more complicated to test such as setup as you have to seed a FormDataCollection with data. On the other hand it's flexible and easy to use and especially with string parameters is easy to deal with. It's also dynamic, so if the client sends you a variety of combinations of values on which you make operating decisions, this is much easier to work with than a strongly typed object that would have to account for all possible values up front.

The downside is that the code looks old school and isn't as self-documenting as a parameter list or object parameter would be. Nevertheless it's totally functionality and a viable choice for collecting POST values.

What about [FromBody]?

Web API also has a [FromBody] attribute that can be assigned to parameters. If you have multiple parameters on a Web API method signature you can use [FromBody] to specify which one will be parsed from the POST content. Unfortunately it's not terribly useful as it only returns content in raw format and requires a totally non-standard format ("=content") to specify your content.

For more info in how FromBody works and several related issues to how POST data is mapped, you can check out Mike Stalls post:

How WebAPI does Parameter Binding

Not really sure where the Web API team thought [FromBody] would really be a good fit other than a down and dirty way to send a full string buffer.

Extending Web API to make multiple POST Vars work? Don't think so

Clearly there's no native support for multiple POST variables being mapped to parameters, which is a bit of a bummer. I know in my own work on one project my customer actually found this to be a real sticking point in their AJAX backend work, and we ended up not using Web API and using MVC JSON features instead. That's kind of sad because Web API is supposed to be the proper solution for AJAX backends.

With all of ASP.NET Web API's extensibility you'd think there would be some way to build this functionality on our own, but after spending a bit of time digging and asking some of the experts from the team and Web API community I didn't hear anything that even suggests that this is possible. From what I could find I'd say it's not possible primarily because Web API's Routing engine does not account for the POST variable mapping. This means [HttpPost] methods with url encoded POST buffers are not mapped to the parameters of the endpoint, and so the routes would never even trigger a request that could be intercepted. Once the routing doesn't work there's not much that can be done.

If somebody has an idea how this could be accomplished I would love to hear about it.

Do we really need multi-value POST mapping?

I think that that POST value mapping is a feature that one would expect of any API tool to have. If you look at common APIs out there like Flicker and Google Maps etc. they all work with POST data. POST data is very prominent much more so than JSON inputs and so supporting as many options that enable would seem to be crucial.

All that aside, Web API does provide very nice features with Model Binding that allows you to capture many POST variables easily enough, and logistically this will let you build whatever you need with POST data of all shapes as long as you map objects. But having to have an object for every operation that receives a data input is going to take its toll in heavy AJAX applications, with a lot of types created that do nothing more than act as parameter containers.

I also think that POST variable mapping is an expected behavior and Web APIs non-support will likely result in many, many questions like this one:

How do I bind a simple POST value in ASP.NET WebAPI RC?

with no clear answer to this question.

I hope for V.next of WebAPI Microsoft will consider this a feature that's worth adding.

Related Articles

© Rick Strahl, West Wind Technologies, 2005-2012
Posted in Web Api  

An Introduction to ASP.NET Web API

$
0
0

Microsoft recently released ASP.NET MVC 4.0 and .NET 4.5 and along with it, the brand spanking new ASP.NET Web API. Web API is an exciting new addition to the ASP.NET stack that provides a new, well-designed HTTP framework for creating REST and AJAX APIs (API is Microsoft’s new jargon for a service, in case you’re wondering). Although Web API ships and installs with ASP.NET MVC 4, you can use Web API functionality in any ASP.NET project, including WebForms, WebPages and MVC or just a Web API by itself. And you can also self-host Web API in your own applications from Console, Desktop or Service applications.

If you're interested in a high level overview on what ASP.NET Web API is and how it fits into the ASP.NET stack you can check out my previous post:

Where does ASP.NET Web API fit?

In the following article, I'll focus on a practical, by example introduction to ASP.NET Web API. All the code discussed in this article is available in GitHub:

https://github.com/RickStrahl/AspNetWebApiArticle

[republished from my Code Magazine Article and updated for RTM release of ASP.NET Web API]

Getting Started

To start I’ll create a new empty ASP.NET application to demonstrate that Web API can work with any kind of ASP.NET project.

Although you can create a new project based on the ASP.NET MVC/Web API template to quickly get up and running, I’ll take you through the manual setup process, because one common use case is to add Web API functionality to an existing ASP.NET application. This process describes the steps needed to hook up Web API to any ASP.NET 4.0 application.

Start by creating an ASP.NET Empty Project. Then create a new folder in the project called Controllers.

Add a Web API Controller Class

Once you have any kind of ASP.NET project open, you can add a Web API Controller class to it. Web API Controllers are very similar to MVC Controller classes, but they work in any kind of project.

Add a new item to this folder by using the Add New Item option in Visual Studio and choose Web API Controller Class, as shown in Figure 1.

Figure 1 -AddNewItemDialog

Figure 1: This is how you create a new Controller Class in Visual Studio

 

Make sure that the name of the controller class includes Controller at the end of it, which is required in order for Web API routing to find it. Here, the name for the class is AlbumApiController.

For this example, I’ll use a Music Album model to demonstrate basic behavior of Web API. The model consists of albums and related songs where an album has properties like Name, Artist and YearReleased and a list of songs with a SongName and SongLength as well as an AlbumId that links it to the album. You can find the code for the model (and the rest of these samples) on Github. To add the file manually, create a new folder called Model, and add a new class Album.cs and copy the code into it. There’s a static AlbumData class with a static CreateSampleAlbumData() method that creates a short list of albums on a static .Current that I’ll use for the examples.

Before we look at what goes into the controller class though, let’s hook up routing so we can access this new controller.

Hooking up Routing in Global.asax

To start, I need to perform the one required configuration task in order for Web API to work: I need to configure routing to the controller. Like MVC, Web API uses routing to provide clean, extension-less URLs to controller methods. Using an extension method to ASP.NET’s static RouteTable class, you can use the MapHttpRoute() (in the System.Web.Http namespace) method to hook-up the routing during Application_Start in global.asax.cs shown in Listing 1.

using System;
using System.Web.Routing;
using System.Web.Http;

namespace AspNetWebApi
{
    public class Global : System.Web.HttpApplication
    {
        protected void Application_Start(object sender, EventArgs e)
        {
            RouteTable.Routes.MapHttpRoute(
                name: "AlbumVerbs",
                routeTemplate: "albums/{title}",
                defaults: new { symbol = RouteParameter.Optional,  
                                controller="AlbumApi" }
            );
        }
    }
}

This route configures Web API to direct URLs that start with an albums folder to the AlbumApiController class. Routing in ASP.NET is used to create extensionless URLs and allows you to map segments of the URL to specific Route Value parameters. A route parameter, with a name inside curly brackets like {name}, is mapped to parameters on the controller methods. Route parameters can be optional, and there are two special route parameters – controller and action – that determine the controller to call and the method to activate respectively.

HTTP Verb Routing

Routing in Web API can route requests by HTTP Verb in addition to standard {controller},{action} routing. For the first examples, I use HTTP Verb routing, as shown Listing 1. Notice that the route I’ve defined does not include an {action} route value or action value in the defaults. Rather, Web API can use the HTTP Verb in this route to determine the method to call the controller, and a GET request maps to any method that starts with Get. So methods called Get() or GetAlbums() are matched by a GET request and a POST request maps to a Post() or PostAlbum(). Web API matches a method by name and parameter signature to match a route, query string or POST values. In lieu of the method name, the [HttpGet,HttpPost,HttpPut,HttpDelete, etc] attributes can also be used to designate the accepted verbs explicitly if you don’t want to follow the verb naming conventions.

Although HTTP Verb routing is a good practice for REST style resource APIs, it’s not required and you can still use more traditional routes with an explicit {action} route parameter. When {action} is supplied, the HTTP verb routing is ignored. I’ll talk more about alternate routes later.

When you’re finished with initial creation of files, your project should look like Figure 2.

 

InitialSolution

Figure 2: The initial project has the new API Controller Album model

 

Creating a small Album Model

Now it’s time to create some controller methods to serve data. For these examples, I’ll use a very simple Album and Songs model to play with, as shown in Listing 2.

public class Song
{
    public string AlbumId { get; set; }
    [Required, StringLength(80)]
    public string SongName { get; set; }
    [StringLength(5)]
    public string SongLength { get; set; }

}

public class Album
{
    public string Id { get; set; }
    [Required, StringLength(80)]
    public string AlbumName { get; set; }
    [StringLength(80)]
    public string Artist { get; set; }
    public int YearReleased { get; set; }
    public DateTime Entered { get; set; }
    [StringLength(150)]
    public string AlbumImageUrl { get; set; }
    [StringLength(200)]
    public string AmazonUrl { get; set; }

    public virtual List<Song> Songs { get; set; }

    public Album()
    {
        Songs = new List<Song>();
        Entered = DateTime.Now;

        // Poor man's unique Id off GUID hash
        Id = Guid.NewGuid().GetHashCode().ToString("x");
    }

    public void AddSong(string songName, string songLength = null)
    {
        this.Songs.Add(new Song()
        {
            AlbumId = this.Id,
            SongName = songName,
            SongLength = songLength
        });
    }
}

Once the model has been created, I also added an AlbumData class that generates some static data in memory that is loaded onto a static .Current member. The signature of this class looks like this and that's what I'll access to retrieve the base data:

public static class AlbumData
{
    // sample data - static list
    public static List<Album> Current = CreateSampleAlbumData();

    /// <summary>
    /// Create some sample data
    /// </summary>
    /// <returns></returns>
    public static List<Album> CreateSampleAlbumData() { … }
}

You can check out the full code for the data generation online.

Creating an AlbumApiController

Web API shares many concepts of ASP.NET MVC, and the implementation of your API logic is done by implementing a subclass of the System.Web.Http.ApiController class. Each public method in the implemented controller is a potential endpoint for the HTTP API, as long as a matching route can be found to invoke it. The class name you create should end in Controller, which is how Web API matches the controller route value to figure out which class to invoke.

Inside the controller you can implement methods that take standard .NET input parameters and return .NET values as results. Web API’s binding tries to match POST data, route values, form values or query string values to your parameters. Because the controller is configured for HTTP Verb based routing (no {action} parameter in the route), any methods that start with Getxxxx() are called by an HTTP GET operation. You can have multiple methods that match each HTTP Verb as long as the parameter signatures are different and can be matched by Web API.

In Listing 3, I create an AlbumApiController with two methods to retrieve a list of albums and a single album by its title .

public class AlbumApiController : ApiController
{

    public IEnumerable<Album> GetAlbums()
    {
        var albums = AlbumData.Current.OrderBy(alb => alb.Artist);
        return albums;
    }
        
    public Album GetAlbum(string title)
    {
        var album = AlbumData.Current
                        .SingleOrDefault(alb => alb.AlbumName.Contains(title));
        return album;
    }
}

To access the first two requests, you can use the following URLs in your browser:

http://localhost/aspnetWebApi/albums
http://localhost/aspnetWebApi/albums/Dirty%20Deeds

Note that you’re not specifying the actions of GetAlbum or GetAlbums in these URLs. Instead Web API’s routing uses HTTP GET verb to route to these methods that start with Getxxx() with the first mapping to the parameterless GetAlbums() method and the latter to the GetAlbum(title) method that receives the title parameter mapped as optional in the route.

Content Negotiation

When you access any of the URLs above from a browser, you get either an XML or JSON result returned back. The album list result for Chrome 17 and Internet Explorer 9 is shown Figure 3.

Figure 3 - StockQuotesOutput

Figure 3: Web API responses can vary depending on the browser used, demonstrating Content Negotiation in action as these two browsers send different HTTP Accept headers.

 

Notice that the results are not the same: Chrome returns an XML response and IE9 returns a JSON response. Whoa, what’s going on here? Shouldn’t we see the same result in both browsers?

Actually, no. Web API determines what type of content to return based on Accept headers. HTTP clients, like browsers, use Accept headers to specify what kind of content they’d like to see returned. Browsers generally ask for HTML first, followed by a few additional content types. Chrome (and most other major browsers) ask for:

Accept: text/html,
application/xhtml+xml,application/xml;
q=0.9,*/*;q=0.8

IE9 asks for:

Accept: text/html, application/xhtml+xml, */*

Note that Chrome’s Accept header includes application/xml, which Web API finds in its list of supported media types and returns an XML response. IE9 does not include an Accept header type that works on Web API by default, and so it returns the default format, which is JSON.

This is an important and very useful feature that was missing from any previous Microsoft REST tools: Web API automatically switches output formats based on HTTP Accept headers. Nowhere in the server code above do you have to explicitly specify the output format. Rather, Web API determines what format the client is requesting based on the Accept headers and automatically returns the result based on the available formatters. This means that a single method can handle both XML and JSON results..

Using this simple approach makes it very easy to create a single controller method that can return JSON, XML, ATOM or even OData feeds by providing the appropriate Accept header from the client. By default you don’t have to worry about the output format in your code.

Note that you can still specify an explicit output format if you choose, either globally by overriding the installed formatters, or individually by returning a lower level HttpResponseMessage instance and setting the formatter explicitly. More on that in a minute.

Along the same lines, any content sent to the server via POST/PUT is parsed by Web API based on the HTTP Content-type of the data sent. The same formats allowed for output are also allowed on input. Again, you don’t have to do anything in your code – Web API automatically performs the deserialization from the content.

Accessing Web API JSON Data with jQuery

A very common scenario for Web API endpoints is to retrieve data for AJAX calls from the Web browser. Because JSON is the default format for Web API, it’s easy to access data from the server using jQuery and its getJSON() method. This example receives the albums array from GetAlbums() and databinds it into the page using knockout.js.

$.getJSON("albums/", function (albums) {
    // make knockout template visible        
    $(".album").show();

    // create view object and attach array   
    var view = { albums: albums };
    ko.applyBindings(view);
});

Figure 4 shows this and the next example’s HTML output. You can check out the complete HTML and script code at http://goo.gl/Ix33C (.html) and http://goo.gl/tETlg (.js).

Figu

Figure 4 -GetAlbumSample

Figure 4: The Album Display sample uses JSON data loaded from Web API.

 

The result from the getJSON() call is a JavaScript object of the server result, which comes back as a JavaScript array. In the code, I use knockout.js to bind this array into the UI, which as you can see, requires very little code, instead using knockout’s data-bind attributes to bind server data to the UI. Of course, this is just one way to use the data – it’s entirely up to you to decide what to do with the data in your client code.

Along the same lines, I can retrieve a single album to display when the user clicks on an album. The response returns the album information and a child array with all the songs. The code to do this is very similar to the last example where we pulled the albums array:

$(".albumlink").live("click", function () {
    var id = $(this).data("id"); // title
    $.getJSON("albums/" + id, function (album) {
        ko.applyBindings(album,
                    $("#divAlbumDialog")[0]);

        $("#divAlbumDialog").show();
    });
});

Here the URL looks like this: /albums/Dirty%20Deeds, where the title is the ID captured from the clicked element’s data ID attribute.

Explicitly Overriding Output Format

When Web API automatically converts output using content negotiation, it does so by matching Accept header media types to the GlobalConfiguration.Configuration.Formatters and the SupportedMediaTypes of each individual formatter. You can add and remove formatters to globally affect what formats are available and it’s easy to create and plug in custom formatters.The example project includes a JSONP formatter that can be plugged in to provide JSONP support for requests that have a callback= querystring parameter. Adding, removing or replacing formatters is a global option you can use to manipulate content. It’s beyond the scope of this introduction to show how it works, but you can review the sample code or check out my blog entry on the subject (http://goo.gl/UAzaR).

If automatic processing is not desirable in a particular Controller method, you can override the response output explicitly by returning an HttpResponseMessage instance. HttpResponseMessage is similar to ActionResult in ASP.NET MVC in that it’s a common way to return an abstract result message that contains content. HttpResponseMessage s parsed by the Web API framework using standard interfaces to retrieve the response data, status code, headers and so on[MS2] . Web API turns every response – including those Controller methods that return static results – into HttpResponseMessage instances. Explicitly returning an HttpResponseMessage instance gives you full control over the output and lets you mostly bypass WebAPI’s post-processing of the HTTP response on your behalf.

HttpResponseMessage allows you to customize the response in great detail. Web API’s attention to detail in the HTTP spec really shows; many HTTP options are exposed as properties and enumerations with detailed IntelliSense comments. Even if you’re new to building REST-based interfaces, the API guides you in the right direction for returning valid responses and response codes.

For example, assume that I always want to return JSON from the GetAlbums() controller method and ignore the default media type content negotiation. To do this, I can adjust the output format and headers as shown in Listing 4.

public HttpResponseMessage GetAlbums()
{
    var albums = AlbumData.Current.OrderBy(alb => alb.Artist);

    // Create a new HttpResponse with Json Formatter explicitly
    var resp = new HttpResponseMessage(HttpStatusCode.OK);
    resp.Content = new ObjectContent<IEnumerable<Album>>(
                            albums, new JsonMediaTypeFormatter());

    // Get Default Formatter based on Content Negotiation
    //var resp = Request.CreateResponse<IEnumerable<Album>>(HttpStatusCode.OK, albums);

    resp.Headers.ConnectionClose = true;
    resp.Headers.CacheControl = new CacheControlHeaderValue();
    resp.Headers.CacheControl.Public = true;

    return resp;
}

This example returns the same IEnumerable<Album> value, but it wraps the response into an HttpResponseMessage so you can control the entire HTTP message result including the headers, formatter and status code. In Listing 4, I explicitly specify the formatter using the JsonMediaTypeFormatter to always force the content to JSON. 

If you prefer to use the default content negotiation with HttpResponseMessage results, you can create the Response instance using the Request.CreateResponse method:

var resp = Request.CreateResponse<IEnumerable<Album>>(HttpStatusCode.OK, albums);

This provides you an HttpResponse object that's pre-configured with the default formatter based on Content Negotiation.

Once you have an HttpResponse object you can easily control most HTTP aspects on this object. What's sweet here is that there are many more detailed properties on HttpResponse than the core ASP.NET Response object, with most options being explicitly configurable with enumerations that make it easy to pick the right headers and response codes from a list of valid codes. It makes HTTP features available much more discoverable even for non-hardcore REST/HTTP geeks.

Non-Serialized Results

The output returned doesn’t have to be a serialized value but can also be raw data, like strings, binary data or streams. You can use the HttpResponseMessage.Content object to set a number of common Content classes. Listing 5 shows how to return a binary image using the ByteArrayContent class from a Controller method.

[HttpGet]
public HttpResponseMessage AlbumArt(string title)
{            
    var album = AlbumData.Current.FirstOrDefault(abl => abl.AlbumName.StartsWith(title));
    if (album == null)
    {
        var resp = Request.CreateResponse<ApiMessageError>(
                                HttpStatusCode.NotFound,
                                new ApiMessageError("Album not found"));
        return resp;
    }

    // kinda silly - we would normally serve this directly  
    // but hey - it's a demo.       
    var http = new WebClient();
    var imageData = http.DownloadData(album.AlbumImageUrl);

    // create response and return 
    var result = new HttpResponseMessage(HttpStatusCode.OK);
    result.Content = new ByteArrayContent(imageData);
    result.Content.Headers.ContentType = new MediaTypeHeaderValue("image/jpeg");

    return result;
}

The image retrieval from Amazon is contrived, but it shows how to return binary data using ByteArrayContent. It also demonstrates that you can easily return multiple types of content from a single controller method, which is actually quite common. If an error occurs - such as a resource can’t be found or a validation error – you can return an error response to the client that’s very specific to the error. In GetAlbumArt(), if the album can’t be found, we want to return a 404 Not Found status (and realistically no error, as it’s an image).

Note that if you are not using HTTP Verb-based routing or not accessing a method that starts with Get/Post etc., you have to specify one or more HTTP Verb attributes on the method explicitly. Here, I used the [HttpGet] attribute to serve the image. Another option to handle the error could be to return a fixed placeholder image if no album could be matched or the album doesn’t have an image.

When returning an error code, you can also return a strongly typed response to the client. For example, you can set the 404 status code and also return a custom error object (ApiMessageError is a class I defined) like this:

return  Request.CreateResponse<ApiMessageError>(
             HttpStatusCode.NotFound,
             new ApiMessageError("Album not found")  
        );            

 

If the album can be found, the image will be returned. The image is downloaded into a byte[] array, and then assigned to the result’s Content property. I created a new ByteArrayContent instance and assigned the image’s bytes and the content type so that it displays properly in the browser.

There are other content classes available: StringContent, StreamContent, ByteArrayContent, MultipartContent, and ObjectContent are at your disposal to return just about any kind of content. You can create your own Content classes if you frequently return custom types and handle the default formatter assignments that should be used to send the data out .

Although HttpResponseMessage results require more code than returning a plain .NET value from a method, it allows much more control over the actual HTTP processing than automatic processing. It also makes it much easier to test your controller methods as you get a response object that you can check for specific status codes and output messages rather than just a result value.

Routing Again

Ok, let’s get back to the image example. Using the original routing we have setup using HTTP Verb routing there's no good way to serve the image. In order to return my album art image I’d like to use a URL like this:

http://localhost/aspnetWebApi/albums/Dirty%20Deeds/image

In order to create a URL like this, I have to create a new Controller because my earlier routes pointed to the AlbumApiController using HTTP Verb routing. HTTP Verb based routing is great for representing a single set of resources such as albums. You can map operations like add, delete, update and read easily using HTTP Verbs. But you cannot mix action based routing into a an HTTP Verb routing controller - you can only map HTTP Verbs and each method has to be unique based on parameter signature. You can't have multiple GET operations to methods with the same signature. So GetImage(string id) and GetAlbum(string title) are in conflict in an HTTP GET routing scenario.

In fact, I was unable to make the above Image URL work with any combination of HTTP Verb plus Custom routing using the single Albums controller.

There are number of ways around this, but all involve additional controllers.  Personally, I think it’s easier to use explicit Action routing and then add custom routes if you need to simplify your URLs further. So in order to accommodate some of the other examples, I created another controller – AlbumRpcApiController – to handle all requests that are explicitly routed via actions (/albums/rpc/AlbumArt) or are custom routed with explicit routes defined in the HttpConfiguration. I added the AlbumArt() method to this new AlbumRpcApiController class.

For the image URL to work with the new AlbumRpcApiController, you need a custom route placed before the default route from Listing 1.

RouteTable.Routes.MapHttpRoute(
    name: "AlbumRpcApiAction",
    routeTemplate: "albums/rpc/{action}/{title}",
    defaults: new
    {
        title = RouteParameter.Optional,
        controller = "AlbumRpcApi",
        action = "GetAblums"
    }
);

Now I can use either of the following URLs to access the image:

Custom route: (/albums/rpc/{title}/image)
http://localhost/aspnetWebApi/albums/PowerAge/image

Action route: (/albums/rpc/action/{title})
http://localhost/aspnetWebAPI/albums/rpc/albumart/PowerAge

Sending Data to the Server

To send data to the server and add a new album, you can use an HTTP POST operation. Since I’m using HTTP Verb-based routing in the original AlbumApiController, I can implement a method called PostAlbum()to accept a new album from the client. Listing 6 shows the Web API code to add a new album.

public HttpResponseMessage PostAlbum(Album album)
{
    if (!this.ModelState.IsValid)
    {
        // my custom error class
        var error = new ApiMessageError() { message = "Model is invalid" };

        // add errors into our client error model for client
        foreach (var prop in ModelState.Values)
        {
            var modelError = prop.Errors.FirstOrDefault();
            if (!string.IsNullOrEmpty(modelError.ErrorMessage))    
                error.errors.Add(modelError.ErrorMessage);
            else
                error.errors.Add(modelError.Exception.Message);
        }

        return Request.CreateResponse<ApiMessageError>(HttpStatusCode.Conflict, error);
    }

    // update song id which isn't provided
    foreach (var song in album.Songs)
        song.AlbumId = album.Id;

    // see if album exists already
    var matchedAlbum = AlbumData.Current
                    .SingleOrDefault(alb => alb.Id == album.Id ||
                                     alb.AlbumName == album.AlbumName);
    if (matchedAlbum == null)
        AlbumData.Current.Add(album);
    else
        matchedAlbum = album;

    // return a string to show that the value got here
    var resp = Request.CreateResponse(HttpStatusCode.OK, string.Empty);
    resp.Content = new StringContent(album.AlbumName + " " + album.Entered.ToString(),
                                        Encoding.UTF8, "text/plain");
    return resp;
}

The PostAlbum() method receives an album parameter, which is automatically deserialized from the POST buffer the client sent. The data passed from the client can be either XML or JSON. Web API automatically figures out what format it needs to deserialize based on the content type and binds the content to the album object. Web API uses model binding to bind the request content to the parameter(s) of controller methods.

Like MVC you can check the model by looking at ModelState.IsValid. If it’s not valid, you can run through the ModelState.Values collection and check each binding for errors. Here I collect the error messages into a string array that gets passed back to the client via the result ApiErrorMessage object.

When a binding error occurs, you’ll want to return an HTTP error response and it’s best to do that with an HttpResponseMessage result. In Listing 6, I used a custom error class that holds a message and an array of detailed error messages for each binding error. I used this object as the content to return to the client along with my Conflict HTTP Status Code response.

If binding succeeds, the example returns a string with the name and date entered to demonstrate that you captured the data. Normally, a method like this should return a Boolean or no response at all (HttpStatusCode.NoConent).

The sample uses a simple static list to hold albums, so once you’ve added the album using the Post operation, you can hit the /albums/ URL to see that the new album was added.

The client jQuery code to call the POST operation from the client with jQuery is shown in Listing 7.

var id = new Date().getTime().toString();
var album = {
    "Id": id,
    "AlbumName": "Power Age",
    "Artist": "AC/DC",
    "YearReleased": 1977,
    "Entered": "2002-03-11T18:24:43.5580794-10:00",
    "AlbumImageUrl": http://ecx.images-amazon.com/images/…,
    "AmazonUrl": http://www.amazon.com/…,
    "Songs": [
        { "SongName": "Rock 'n Roll Damnation", "SongLength": 3.12},
        { "SongName": "Downpayment Blues", "SongLength": 4.22 },
        { "SongName": "Riff Raff", "SongLength": 2.42 }
    ]
}

$.ajax(
{
    url: "albums/",
    type: "POST",
    contentType: "application/json",
    data: JSON.stringify(album),
    processData: false,
    beforeSend: function (xhr) {
        // not required since JSON is default output
        xhr.setRequestHeader("Accept", "application/json");
    },
    success: function (result) {
        // reload list of albums
        page.loadAlbums();
    },
    error: function (xhr, status, p3, p4) {
        var err = "Error";
        if (xhr.responseText && xhr.responseText[0] == "{")
            err = JSON.parse(xhr.responseText).message;
        alert(err);
    }
});

The code in Listing 7 creates an album object in JavaScript to match the structure of the .NET Album class. This object is passed to the $.ajax() function to send to the server as POST. The data is turned into JSON and the content type set to application/json so that the server knows what to convert when deserializing in the Album instance.

The jQuery code hooks up success and failure events. Success returns the result data, which is a string that’s echoed back with an alert box. If an error occurs, jQuery returns the XHR instance and status code. You can check the XHR to see if a JSON object is embedded and if it is, you can extract it by de-serializing it and accessing the .message property.

REST standards suggest that updates to existing resources should use PUT operations. REST standards aside, I’m not a big fan of separating out inserts and updates so I tend to have a single method that handles both.

But if you want to follow REST suggestions, you can create a PUT method that handles updates by forwarding the PUT operation to the POST method:

public HttpResponseMessage PutAlbum(Album album)
{
     return PostAlbum(album);
}

To make the corresponding $.ajax() call, all you have to change from Listing 7 is the type: from POST to PUT.

Model Binding with UrlEncoded POST Variables

In the example in Listing 7 I used JSON objects to post a serialized object to a server method that accepted an strongly typed object with the same structure, which is a common way to send data to the server. However, Web API supports a number of different ways that data can be received by server methods.

For example, another common way is to use plain UrlEncoded POST  values to send to the server. Web API supports Model Binding that works similar (but not the same) as MVC's model binding where POST variables are mapped to properties of object parameters of the target method. This is actually quite common for AJAX calls that want to avoid serialization and the potential requirement of a JSON parser on older browsers.

For example, using jQUery you might use the $.post() method to send a new album to the server (albeit one without songs) using code like the following:

$.post("albums/",{AlbumName: "Dirty Deeds", YearReleased: 1976 … },albumPostCallback);

Although the code looks very similar to the client code we used before passing JSON, here the data passed is URL encoded values (AlbumName=Dirty+Deeds&YearReleased=1976 etc.). Web API then takes this POST data and maps each of the POST values to the properties of the Album object in the method's parameter. Although the client code is different the server can both handle the JSON object, or the UrlEncoded POST values.

Dynamic Access to POST Data

There are also a few options available to dynamically access POST data, if you know what type of data you're dealing with.

If you have POST UrlEncoded values, you can dynamically using a FormsDataCollection:

[HttpPost]
public string PostAlbum(FormDataCollection form)
{
    return string.Format("{0} - released {1}",
                            form.Get("AlbumName"),form.Get("RearReleased"));                   
}

The FormDataCollection is a very simple object, that essentially provides the same functionality as Request.Form[] in ASP.NET. Request.Form[] still works if you're running hosted in an ASP.NET application. However as a general rule, while ASP.NET's functionality is always available when running Web API hosted inside of an  ASP.NET application, using the built in classes specific to Web API makes it possible to run Web API applications in a self hosted environment outside of ASP.NET.

If your client is sending JSON to your server, and you don't want to map the JSON to a strongly typed object because you only want to retrieve a few simple values, you can also accept a JObject parameter in your API methods:

[HttpPost]
public string PostAlbum(JObject jsonData)
{
    dynamic json = jsonData;
    JObject jalbum = json.Album;
    JObject juser = json.User;
    string token = json.UserToken;

    var album = jalbum.ToObject<Album>();
    var user = juser.ToObject<User>();

    return String.Format("{0} {1} {2}", album.AlbumName, user.Name, token);
}

There quite a few options available to you to receive data with Web API, which gives you more choices for the right tool for the job.

Unfortunately one shortcoming of Web API is that POST data is always mapped to a single parameter. This means you can't pass multiple POST parameters to methods that receive POST data. It's possible to accept multiple parameters, but only one can map to the POST content - the others have to come from the query string or route values.

I have a couple of Blog POSTs that explain what works and what doesn't here:

 

Handling Delete Operations

Finally, to round out the server API code of the album example we've been discussin, here’s the DELETE verb controller method that allows removal of an album by its title:

public HttpResponseMessage DeleteAlbum(string title)
{
    var matchedAlbum = AlbumData.Current.Where(alb => alb.AlbumName == title)
                                    .SingleOrDefault();
    if (matchedAlbum == null)
        return new HttpResponseMessage(HttpStatusCode.NotFound);

    AlbumData.Current.Remove(matchedAlbum);

    return new HttpResponseMessage(HttpStatusCode.NoContent);
}

To call this action method using jQuery, you can use:

$(".removeimage").live("click", function () {
    var $el = $(this).parent(".album");
    var txt = $el.find("a").text();
    $.ajax({
        url: "albums/" + encodeURIComponent(txt),
        type: "Delete",
        success: function (result) {
            $el.fadeOut().remove();
        },
        error: jqError
    });
}

 

Note the use of the DELETE verb in the $.ajax() call, which routes to DeleteAlbum on the server. DELETE is a non-content operation, so you supply a resource ID (the title) via route value or the querystring.

Routing Conflicts

In all requests with the exception of the AlbumArt image example shown so far, I used HTTP Verb routing that I set up in Listing 1. HTTP Verb Routing is a recommendation that is in line with typical REST access to HTTP resources. However, it takes quite a bit of effort to create REST-compliant API implementations based only on HTTP Verb routing only. You saw one example that didn’t really fit – the return of an image where I created a custom route albums/{title}/image that required creation of a second controller and a custom route to work. HTTP Verb routing to a controller does not mix with custom or action routing to the same controller because of the limited mapping of HTTP verbs imposed by HTTP Verb routing.

To understand some of the problems with verb routing, let’s look at another example. Let’s say you create a GetSortableAlbums() method like this and add it to the original AlbumApiController accessed via HTTP Verb routing:

[HttpGet]
public IQueryable<Album> SortableAlbums()
{
    var albums = AlbumData.Current;
         
    // generally should be done only on actual queryable results (EF etc.)
    // Done here because we're running with a static list but otherwise might be slow
    return albums.AsQueryable();
}

If you compile this code and try to now access the /albums/ link, you get an error: Multiple Actions were found that match the request.

HTTP Verb routing only allows access to one GET operation per parameter/route value match. If more than one method exists with the same parameter signature, it doesn’t work. As I mentioned earlier for the image display, the only solution to get this method to work is to throw it into another controller. Because I already set up the AlbumRpcApiController I can add the method there.

First, I should rename the method to SortableAlbums() so I’m not using a Get prefix for the method. This also makes the action parameter look cleaner in the URL - it looks less like a method and more like a noun.

I can then create a new route that handles direct-action mapping:

RouteTable.Routes.MapHttpRoute(
    name: "AlbumRpcApiAction",
    routeTemplate: "albums/rpc/{action}/{title}",
    defaults: new
    {
        title = RouteParameter.Optional,
        controller = "AlbumRpcApi",
        action = "GetAblums"
    }
);

As I am explicitly adding a route segment – rpc – into the route template, I can now reference explicit methods in the Web API controller using URLs like this:

http://localhost/AspNetWebApi/rpc/SortableAlbums

Error Handling

I’ve already done some minimal error handling in the examples. For example in Listing 6, I detected some known-error scenarios like model validation failing or a resource not being found and returning an appropriate HttpResponseMessage result. But what happens if your code just blows up or causes an exception?

If you have a controller method, like this:

[HttpGet]
public void ThrowException()
{
    throw new UnauthorizedAccessException("Unauthorized Access Sucka");
}

You can call it with this:

http://localhost/AspNetWebApi/albums/rpc/ThrowException

The default exception handling displays a 500-status response with the serialized exception on the local computer only. When you connect from a remote computer, Web API throws back a 500  HTTP Error with no data returned (IIS then adds its HTML error page). The behavior is configurable in the GlobalConfiguration:

GlobalConfiguration
        .Configuration
        .IncludeErrorDetailPolicy = IncludeErrorDetailPolicy.Never; 

If you want more control over your error responses sent from code, you can throw explicit error responses yourself using HttpResponseException. When you throw an HttpResponseException the response parameter is used to generate the output for the Controller action.

[HttpGet]
public void ThrowError()
{

    var resp = Request.CreateResponse<ApiMessageError>(
            HttpStatusCode.BadRequest,
            new ApiMessageError("Your code stinks!"));
    throw new HttpResponseException(resp);
}

Throwing an HttpResponseException stops the processing of the controller method and immediately returns the response you passed to the exception. Unlike other Exceptions fired inside of WebAPI, HttpResponseException bypasses the Exception Filters installed and instead just outputs the response you provide.

In this case, the serialized ApiMessageError result string is returned in the default serialization format – XML or JSON. You can pass any content to HttpResponseMessage, which includes creating your own exception objects and consistently returning error messages to the client. Here’s a small helper method on the controller that you might use to send exception info back to the client consistently:

private void ThrowSafeException(string message,
            HttpStatusCode statusCode = HttpStatusCode.BadRequest)
{
    var errResponse = Request.CreateResponse<ApiMessageError>(statusCode,
                                new ApiMessageError() { message = message });

    throw new HttpResponseException(errResponse);
}

You can then use it to output any captured errors from code:

[HttpGet]
public void ThrowErrorSafe()
{
    try
    {
        List<string> list = null;
        list.Add("Rick");
    }
    catch (Exception ex)
    { ThrowSafeException(ex.Message); }
}

 

Exception Filters

Another more global solution is to create an Exception Filter. Filters in Web API provide the ability to pre- and post-process controller method operations. An exception filter looks at all exceptions fired and then optionally creates an HttpResponseMessage result. Listing 8 shows an example of a basic Exception filter implementation.

public class UnhandledExceptionFilter : ExceptionFilterAttribute
{
    public override void OnException(HttpActionExecutedContext context)
    {
        HttpStatusCode status = HttpStatusCode.InternalServerError;

        var exType = context.Exception.GetType();

        if (exType == typeof(UnauthorizedAccessException))
            status = HttpStatusCode.Unauthorized;
        else if (exType == typeof(ArgumentException))
            status = HttpStatusCode.NotFound;

        var apiError = new ApiMessageError() { message = context.Exception.Message };

        // create a new response and attach our ApiError object
        // which now gets returned on ANY exception result
        var errorResponse = context.Request.CreateResponse<ApiMessageError>(status, apiError);
        //var errorResponse = context.Request.CreateResponse(HttpStatusCode.BadRequest, 
        //                                context.Exception.GetBaseException().Message);
        context.Response = errorResponse;

        base.OnException(context);
    }
}

Exception Filter Attributes can be assigned to an ApiController class like this:

[UnhandledExceptionFilter]
public class AlbumRpcApiController : ApiController

or you can globally assign it to all controllers by adding it to the HTTP Configuration's Filters collection:

GlobalConfiguration.Configuration.Filters.Add(new UnhandledExceptionFilter());

The latter is a great way to get global error trapping so that all errors (short of hard IIS errors and explicit HttpResponseException errors) return a valid error response that includes error information in the form of a known-error object. Using a filter like this allows you to throw an exception as you normally would and have your filter create a response in the appropriate output format that the client expects. For example, an AJAX application can on failure expect to see a JSON error result that corresponds to the real error that occurred rather than a 500 error along with HTML error page that IIS throws up.

You can even create some custom exceptions so you can differentiate your own exceptions from unhandled system exceptions - you often don't want to display error information from 'unknown' exceptions as they may contain sensitive system information or info that's not generally useful to users of your application/site.

This is just one example of how ASP.NET Web API is configurable and extensible. Exception filters are just one example of how you can plug-in into the Web API request flow to modify output. Many more hooks exist and I’ll take a closer look at extensibility in Part 2 of this article in the future.

Summary

Web API is a big improvement over previous Microsoft REST and AJAX toolkits. The key features to its usefulness are its ease of use with simple controller based logic, familiar MVC-style routing, low configuration impact, extensibility at all levels and tight attention to exposing and making HTTP semantics easily discoverable and easy to use. Although none of the concepts used in Web API are new or radical, Web API combines the best of previous platforms into a single framework that’s highly functional, easy to work with, and extensible to boot. I think that Microsoft has hit a home run with Web API.

Related Resources

© Rick Strahl, West Wind Technologies, 2005-2012
Posted in Web Api  

.NET 3.5 Installation Problems in Windows 8

$
0
0

Windows 8 installs with .NET 4.5. A default installation of Windows 8 doesn't seem to include .NET 3.0 or 3.5, although .NET 2.0 does seem to be available by default (presumably because Windows has app dependencies on that). I ran into some pretty nasty compatibility issues regarding .NET 3.5 which I'll describe in this post.

I'll preface this by saying that depending on how you install Windows 8 you may not run into these issues. In fact, it's probably a special case, but one that might be common with developer folks reading my blog. Specifically it's the install order that screwed things up for me -  installing Visual Studio before explicitly installing .NET 3.5 from Windows Features - in particular. If you install Visual Studio 2010 I highly recommend you install .NET 3.5 from Windows features BEFORE you install Visual Studio 2010 and save yourself the trouble I went through.

So when I installed Windows 8, and then looked at the Windows Features to install after the fact in the Windows Feature dialog, I thought - .NET 3.5 - who needs it. I'd be happy to not have to install .NET 3.5, but unfortunately I found out quite a while after initial installation that one of my applications/tools (DevExpress's awesome CodeRush) depends on it and won't install without it.

Enabling .NET 3.5 in Windows 8

If you want to run .NET 3.5 on Windows 8, don't download an installer - those installers don't work on Windows 8, and you don't need to do this because you can use the Windows Features dialog to enable .NET 3.5:

WindowsFeatures

And that *should* do the trick. If you do this before you install other apps that require .NET 3.5 and install a non-SP1 one version of it, you are going to have no problems.

Unfortunately for me, even after I've installed the above, when I run the CodeRush installer I still get this lovely dialog:

DotNet35NotInstalled

Now I double checked to see if .NET 3.5 is installed - it is, both for 32 bit and 64 bit. I went as far as creating a small .NET Console app and running it to verify that it actually runs. And it does…

So naturally I thought the CodeRush installer is a little whacky.

After some back and forth Alex Skorkin on Twitter pointed me in the right direction: He asked me to look in the registry for exact info on which version of .NET 3.5 is installed here:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\NET Framework Setup\NDP

where I found that .NET 3.5 SP1 was installed. This is the 64 bit key which looks all correct.

However, when I looked under the 32 bit node I found:

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\NET Framework Setup\NDP\v3.5

RegEdit

Notice that the service pack number is set to 0, rather than 1 (which it was for the 64 bit install), which is what the installer requires.

So to summarize: the 64 bit version is installed with SP1, the 32 bit version is not. Uhm, Ok… thanks for that!

Easy to fix, you say - just install SP1. Nope, not so easy because the standalone installer doesn't work on Windows 8. I can't get either .NET 3.5 installer or the SP 1 installer to even launch. They simply start and hang (or exit immediately) without messages.

I also tried to get Windows to update .NET 3.5 by checking for Windows Updates, which should pick up on the dated version of .NET 3.5 and pull down SP1, but that's also no go. Check for Updates doesn't bring down any updates for me yet. I'm sure at some random point in the future Windows will deem it necessary to update .NET 3.5 to SP1, but at this point it's not letting me coerce it to do it explicitly.

How did this happen

I'm not sure exactly whether this is the cause and effect, but I suspect the story goes like this:

  1. Installed Windows 8 without support for .NET 3.5
  2. Installed Visual Studio 2010 which installs .NET 3.5 (no SP)

I now had .NET 3.5 installed but without SP1. I then:

  1. Tried to install CodeRush - Error: .NET 3.5 SP1 required
  2. Enabled .NET 3.5 in Windows Features

I figured enabling the .NET 3.5 Windows Features would do the trick. But still no go.

Now I suspect Visual Studio installed the 32 bit version of .NET 3.5 on my machine and Windows Features detected the previous install and didn't reinstall it. This left the 32 bit install at least with no SP1 installed.

How to Fix it

My final solution was to completely uninstall .NET 3.5 *and* to reboot:

  1. Go to Windows Features
  2. Uncheck the .NET Framework 3.5
  3. Restart Windows
  4. Go to Windows Features
  5. Check .NET Framework 3.5

and voila, I now have a proper installation of .NET 3.5.

I tried this before but without the reboot step in between which did not work. Make sure you reboot between uninstalling and reinstalling .NET 3.5!

More Problems

The above fixed me right up, but in looking for a solution it seems that a lot of people are also having problems with .NET 3.5 installing properly from the Windows Features dialog. The problem there is that the feature wasn't properly loading from the installer disks or not downloading the proper components for updates.

It turns out you can explicitly install Windows features using the DISM tool in Windows.

dism.exe /online /enable-feature /featurename:NetFX3 /Source:f:\sources\sxs 

You can try this without the /Source flag first - which uses the hidden Windows installer files if you kept those. Otherwise insert the DVD or ISO and point at the path \sources\sxs path where the installer lives.

This also gives you a little more information if something does go wrong.

© Rick Strahl, West Wind Technologies, 2005-2012
Posted in Windows  .NET  

Using JSON.NET for dynamic JSON parsing

$
0
0

With the release of ASP.NET Web API as part of .NET 4.5 and MVC 4.0, JSON.NET has effectively pushed out the .NET native serializers to become the default serializer for Web API. JSON.NET is vastly more flexible than the built in DataContractJsonSerializer or the older JavaScript serializer. The DataContractSerializer in particular has been very problematic in the past because it can't deal with untyped objects for serialization - like values of type object, or anonymous types which are quite common these days. The JavaScript Serializer that came before it actually does support non-typed objects for serialization but it can't do anything with untyped data coming in from JavaScript and it's overall model of extensibility was pretty limited (JavaScript Serializer is what MVC uses for JSON responses).

JSON.NET provides a robust JSON serializer that has both high level and low level components, supports binary JSON, JSON contracts, Xml to JSON conversion, LINQ to JSON and many, many more features than either of the built in serializers. ASP.NET Web API now uses JSON.NET as its default serializer and is now pulled in as a NuGet dependency into Web API projects, which is great.

Dynamic JSON Parsing

One of the features that I think is getting ever more important is the ability to serialize and deserialize arbitrary JSON content dynamically - that is without mapping the JSON captured directly into a .NET type as DataContractSerializer or the JavaScript Serializers do. Sometimes it isn't possible to map types due to the differences in languages (think collections, dictionaries etc), and other times you simply don't have the structures in place or don't want to create them to actually import the data.

If this topic sounds familiar - you're right! I wrote about dynamic JSON parsing a few months back before JSON.NET was added to Web API and when Web API and the System.Net HttpClient libraries included the System.Json classes like JsonObject and JsonArray. With the inclusion of JSON.NET in Web API these classes are now obsolete and didn't ship with Web API or the client libraries. I re-linked my original post to this one. In this post I'll discus JToken, JObject and JArray which are the dynamic JSON objects that make it very easy to create and retrieve JSON content on the fly without underlying types.

Why Dynamic JSON?

So, why Dynamic JSON parsing rather than strongly typed parsing? Since applications are interacting more and more with third party services it becomes ever more important to have easy access to those services with easy JSON parsing. Sometimes it just makes lot of sense to pull just a small amount of data out of large JSON document received from a service, because the third party service isn't directly related to your application's logic most of the time - and it makes little sense to map the entire service structure in your application.

For example, recently I worked with the Google Maps Places API to return information about businesses close to me (or rather the app's) location. The Google API returns a ton of information that my application had no interest in - all I needed was few values out of the data. Dynamic JSON parsing makes it possible to map this data, without having to map the entire API to a C# data structure. Instead I could pull out the three or four values I needed from the API and directly store it on my business entities that needed to receive the data - no need to map the entire Maps API structure.

Getting JSON.NET

The easiest way to use JSON.NET is to grab it via NuGet and add it as a reference to your project. You can add it to your project with:

PM> Install-Package Newtonsoft.Json

From the Package Manager Console or by using Manage NuGet Packages in your project References. As mentioned if you're using ASP.NET Web API or MVC 4 JSON.NET will be automatically added to your project.

Alternately you can also go to the CodePlex site and download the latest version including source code:

http://json.codeplex.com/

Creating JSON on the fly with JObject and JArray

Let's start with creating some JSON on the fly. It's super easy to create a dynamic object structure with any of the JToken derived JSON.NET objects. The most common JToken derived classes you are likely to use are JObject and JArray.

JToken implements IDynamicMetaProvider and so uses the dynamic  keyword extensively to make it intuitive to create object structures and turn them into JSON via dynamic object syntax. Here's an example of creating a music album structure with child songs using JObject for the base object and songs and JArray for the actual collection of songs:

[TestMethod]
public void JObjectOutputTest()
{
    // strong typed instance 
    var jsonObject = new JObject();
                        
    // you can explicitly add values here using class interface
    jsonObject.Add("Entered", DateTime.Now);

    // or cast to dynamic to dynamically add/read properties
    dynamic album = jsonObject;

    album.AlbumName = "Dirty Deeds Done Dirt Cheap";
    album.Artist = "AC/DC";
    album.YearReleased = 1976;

    album.Songs = new JArray() as dynamic;
            
    dynamic song = new JObject();
    song.SongName = "Dirty Deeds Done Dirt Cheap";
    song.SongLength = "4:11";
    album.Songs.Add(song);

    song = new JObject();
    song.SongName = "Love at First Feel";
    song.SongLength = "3:10";
    album.Songs.Add(song);

    Console.WriteLine(album.ToString());
}

This produces a complete JSON structure:

{
  "Entered": "2012-08-18T13:26:37.7137482-10:00",
  "AlbumName": "Dirty Deeds Done Dirt Cheap",
  "Artist": "AC/DC",
  "YearReleased": 1976,
  "Songs": [
    {
      "SongName": "Dirty Deeds Done Dirt Cheap",
      "SongLength": "4:11"
    },
    {
      "SongName": "Love at First Feel",
      "SongLength": "3:10"
    }
  ]
}

Notice that JSON.NET does a nice job formatting the JSON, so it's easy to read and paste into blog posts :-). JSON.NET includes a bunch of configuration options that control how JSON is generated. Typically the defaults are just fine, but you can override with the JsonSettings object for most operations.

The important thing about this code is that there's no explicit type used for holding the values to serialize to JSON. Rather the JSON.NET objects are the containers that receive the data as I build up my JSON structure dynamically, simply by adding properties. This means this code can be entirely driven at runtime without compile time restraints of structure for the JSON output.

Here I use JObject to create a album 'object' and immediately cast it to dynamic. JObject() is kind of similar in behavior to ExpandoObject in that it allows you to add properties by simply assigning to them. Internally, JObject values are stored in pseudo collections of key value pairs that are exposed as properties through the IDynamicMetaObject interface exposed in JSON.NET's JToken base class.

For objects the syntax is very clean - you add simple typed values as properties. For objects and arrays you have to explicitly create new JObject or JArray, cast them to dynamic and then add properties and items to them. Always remember though these values are dynamic - which means no Intellisense and no compiler type checking. It's up to you to ensure that the names and values you create are accessed consistently and without typos in your code.

Note that you can also access the JObject instance directly (not as dynamic) and get access to the underlying JObject type. This means you can assign properties by string, which can be useful for fully data driven JSON generation from other structures.

Below you can see both styles of access next to each other:

// strong type instance 
var jsonObject = new JObject();

// you can explicitly add values here
jsonObject.Add("Entered", DateTime.Now);

// expando style instance you can just 'use' properties
dynamic album = jsonObject;

album.AlbumName = "Dirty Deeds Done Dirt Cheap";

JContainer (the base class for JObject and JArray) is a collection so you can also iterate over the properties at runtime easily:

foreach (var item in jsonObject)
{
    Console.WriteLine(item.Key + " " + item.Value.ToString());
}

The functionality of the JSON objects are very similar to .NET's ExpandObject and if you used it before, you're already familiar with how the dynamic interfaces to the JSON objects works.

Importing JSON with JObject.Parse() and JArray.Parse()

The JValue structure supports importing JSON via the Parse() and Load() methods which can read JSON data from a string or various streams respectively. Essentially JValue includes the core JSON parsing to turn a JSON string into a collection of JsonValue objects that can be then referenced using familiar dynamic object syntax.

Here's a simple example:

public void JValueParsingTest()
{
    var jsonString = @"{""Name"":""Rick"",""Company"":""West Wind"",
""Entered"":""2012-03-16T00:03:33.245-10:00""}"
; dynamic json = JValue.Parse(jsonString); // values require casting string name = json.Name; string company = json.Company; DateTime entered = json.Entered; Assert.AreEqual(name, "Rick"); Assert.AreEqual(company, "West Wind"); }

The JSON string represents an object with three properties which is parsed into a JObject class and cast to dynamic. Once cast to dynamic I can then go ahead and access the object using familiar object syntax.

Note that the actual values - json.Name, json.Company, json.Entered - are actually of type JToken and I have to cast them to their appropriate types first before I can do type comparisons as in the Asserts at the end of the test method. This is required because of the way that dynamic types work which can't determine the type based on the method signature of the Assert.AreEqual(object,object) method. I have to either assign the dynamic value to a variable as I did above, or explicitly cast ( (string) json.Name) in the actual method call.

The JSON structure can be much more complex than this simple example. Here's another example of an array of albums serialized to JSON and then parsed through with JsonValue():

[TestMethod]
public void JsonArrayParsingTest()
{
    var jsonString = @"[
{
""Id"": ""b3ec4e5c"",
""AlbumName"": ""Dirty Deeds Done Dirt Cheap"",
""Artist"": ""AC/DC"",
""YearReleased"": 1976,
""Entered"": ""2012-03-16T00:13:12.2810521-10:00"",
""AlbumImageUrl"": ""http://ecx.images-amazon.com/images/I/61kTaH-uZBL._AA115_.jpg"",
""AmazonUrl"": ""http://www.amazon.com/gp/product/…ASIN=B00008BXJ4"",
""Songs"": [
    {
    ""AlbumId"": ""b3ec4e5c"",
    ""SongName"": ""Dirty Deeds Done Dirt Cheap"",
    ""SongLength"": ""4:11""
    },
    {
    ""AlbumId"": ""b3ec4e5c"",
    ""SongName"": ""Love at First Feel"",
    ""SongLength"": ""3:10""
    },
    {
    ""AlbumId"": ""b3ec4e5c"",
    ""SongName"": ""Big Balls"",
    ""SongLength"": ""2:38""
    }
]
},
{
""Id"": ""7b919432"",
""AlbumName"": ""End of the Silence"",
""Artist"": ""Henry Rollins Band"",
""YearReleased"": 1992,
""Entered"": ""2012-03-16T00:13:12.2800521-10:00"",
""AlbumImageUrl"": ""http://ecx.images-amazon.com/images/I/51FO3rb1tuL._SL160_AA160_.jpg"",
""AmazonUrl"": ""http://www.amazon.com/End-Silence-Rollins-Band/dp/B0000040OX/ref=sr_1_5?ie=UTF8&qid=1302232195&sr=8-5"",
""Songs"": [
    {
    ""AlbumId"": ""7b919432"",
    ""SongName"": ""Low Self Opinion"",
    ""SongLength"": ""5:24""
    },
    {
    ""AlbumId"": ""7b919432"",
    ""SongName"": ""Grip"",
    ""SongLength"": ""4:51""
    }
]
}
]";

    JArray jsonVal = JArray.Parse(jsonString) as JArray;
    dynamic albums = jsonVal;

    foreach (dynamic album in albums)
    {
        Console.WriteLine(album.AlbumName + " (" + album.YearReleased.ToString() + ")");
        foreach (dynamic song in album.Songs)
        {
            Console.WriteLine("\t" + song.SongName);
        }
    }

    Console.WriteLine(albums[0].AlbumName);
    Console.WriteLine(albums[0].Songs[1].SongName);
}

JObject and JArray in ASP.NET Web API

Of course these types also work in ASP.NET Web API controller methods. If you want you can accept parameters using these object or return them back to the server.

The following contrived example receives dynamic JSON input, and then creates a new dynamic JSON object and returns it based on data from the first:

[HttpPost]
public JObject PostAlbumJObject(JObject jAlbum)
{
    // dynamic input from inbound JSON
    dynamic album = jAlbum;

    // create a new JSON object to write out
    dynamic newAlbum = new JObject();

    // Create properties on the new instance
    // with values from the first
    newAlbum.AlbumName = album.AlbumName + " New";
    newAlbum.NewProperty = "something new";
    newAlbum.Songs = new JArray();
    
    foreach (dynamic song in album.Songs)
    {
        song.SongName = song.SongName + " New"; 
        newAlbum.Songs.Add(song);                
    }
            
    return newAlbum;
}

The raw POST request to the server looks something like this:

POST http://localhost/aspnetwebapi/samples/PostAlbumJObject HTTP/1.1
User-Agent: Fiddler
Content-type: application/json
Host: localhost
Content-Length: 88

{AlbumName: "Dirty Deeds",Songs:[ { SongName: "Problem Child"},{ SongName: "Squealer"}]}

and the output that comes back looks like this:

{
  "AlbumName": "Dirty Deeds New",
  "NewProperty": "something new",
  "Songs": [
    {
      "SongName": "Problem Child New"
    },
    {
      "SongName": "Squealer New"
    }
  ]
}

The original values are echoed back with something extra appended to demonstrate that we're working with a new object.

When you receive or return a JObject, JValue, JToken or JArray instance in a Web API method, Web API ignores normal content negotiation and assumes your content is going to be received and returned as JSON, so effectively the parameter and result type explicitly determines the input and output format which is nice.

Dynamic to Strong Type Mapping

You can also map JObject and JArray instances to a strongly typed object, so you can mix dynamic and static typing in the same piece of code. Using the 2 Album jsonString shown earlier, the code below takes an array of albums and picks out only a single album and casts that album to a static Album instance.

[TestMethod]
public void JsonParseToStrongTypeTest()
{
    JArray albums = JArray.Parse(jsonString) as JArray;

    // pick out one album
    JObject jalbum = albums[0] as JObject;

    // Copy to a static Album instance
    Album album = jalbum.ToObject<Album>();

    Assert.IsNotNull(album);
    Assert.AreEqual(album.AlbumName,jalbum.Value<string>("AlbumName"));
    Assert.IsTrue(album.Songs.Count > 0);
}

This is pretty damn useful for the scenario I mentioned earlier - you can read a large chunk of JSON and dynamically walk the property hierarchy down to the item you want to access, and then either access the specific item dynamically (as shown earlier) or map a part of the JSON to a strongly typed object. That's very powerful if you think about it - it leaves you in total control to decide what's dynamic and what's static.

Strongly typed JSON Parsing

With all this talk of dynamic let's not forget that JSON.NET of course also does strongly typed serialization which is drop dead easy. Here's a simple example on how to serialize and deserialize an object with JSON.NET:

[TestMethod]
public void StronglyTypedSerializationTest()
{

    // Demonstrate deserialization from a raw string
    var album = new Album()
    {
        AlbumName = "Dirty Deeds Done Dirt Cheap",
        Artist = "AC/DC",
        Entered = DateTime.Now,
        YearReleased = 1976,
        Songs =  new List<Song>() 
        {
            new Song()
            {
                SongName = "Dirty Deeds Done Dirt Cheap",
                SongLength = "4:11"
            },
            new Song()
            {
                SongName = "Love at First Feel",
                SongLength = "3:10"
            }
        }
    };
            
    // serialize to string            
    string json2 = JsonConvert.SerializeObject(album,Formatting.Indented);

    Console.WriteLine(json2);

    // make sure we can serialize back
    var album2 = JsonConvert.DeserializeObject<Album>(json2);

    Assert.IsNotNull(album2);
    Assert.IsTrue(album2.AlbumName == "Dirty Deeds Done Dirt Cheap");
    Assert.IsTrue(album2.Songs.Count == 2);
}

JsonConvert is a high level static class that wraps lower level functionality, but you can also use the JsonSerializer class, which allows you to serialize/parse to and from streams. It's a little more work, but gives you a bit more control. The functionality available is easy to discover with Intellisense, and that's good because there's not a lot in the way of documentation that's actually useful.

Summary

JSON.NET is a pretty complete JSON implementation with lots of different choices for JSON parsing from dynamic parsing to static serialization, to complex querying of JSON objects using LINQ. It's good to see this open source library getting integrated into .NET, and pushing out the old and tired stock .NET parsers so that we finally have a bit more flexibility - and extensibility - in our JSON parsing. Good to go!

Resources

© Rick Strahl, West Wind Technologies, 2005-2012
Posted in .NET  Web Api  AJAX  

ASP.NET Frameworks and Raw Throughput Performance

$
0
0

A few days ago I had a curious thought: With all these different technologies that the ASP.NET stack has to offer, what's the fastest technology overall to return raw data for a server request? When I started this it was mere curiosity rather than a real practical need or result. Different tools are used for different problems and so performance differences are to be expected. But still I was curious to see how the various technologies performed relative to each orher just for raw throughput of the request getting to the endpoint and back out to the client with as little processing in the actual endpoint logic as possible  (aka Hello World).

I want to clarify that this is merely an informal test for my own curiosity and I'm sharing the results and process here because I thought it was interesting. It's been a long while since I've done any sort of perf testing on ASP.NET, mainly because I've not had extremely heavy load requirements and because overall ASP.NET performs very well even for fairly high loads so that often it's not that critical to test for load performance. And that's a good thing!

This post is not meant to make a point  or even come to a conclusion which tech is better, but just to act as a reference to help understand some of the differences in perf and give a starting point to play around with yourself. I've included the code for this simple project, so you can play with it and maybe add a few additional tests for different things if you like:

I looked at these technologies:

  1. ASP.NET Web API
  2. ASP.NET MVC
  3. WebForms
  4. ASP.NET WebPages
  5. ASMX AJAX Services  (couldn't get AJAX/JSON to run on IIS8 )
  6. WCF Rest
  7. Raw ASP.NET HttpHandlers

It's quite a mixed bag, of course and the technologies target different types of development. What started out as mere curiosity turned into a bit of a head scratcher as the results were sometimes surprising.

First test: Raw Throughput

The first thing I tested was raw throughput for the various technologies. This is the least practical test of course since you're unlikely to ever create the equivalent of a 'Hello World' request in a real life application. The idea here is to measure how much time a 'NOP' request takes to return data to the client. So for this request I create the simplest Hello World request that I could come up for each tech.

Http Handler

The first is the lowest level approach which is an HTTP handler.

    public class Handler : IHttpHandler
    {
        public void ProcessRequest(HttpContext context)
        {
            context.Response.ContentType = "text/plain";
            context.Response.Write("Hello World. Time is: " + DateTime.Now.ToString());
        }

        public bool IsReusable
        {
            get
            {
                return true;
            }
        }
    }

WebForms

Next I added a couple of ASPX pages - one using CodeBehind and one using only a markup page.

The CodeBehind page simple does this in CodeBehind without any markup in the ASPX page:

    public partial class HelloWorld_CodeBehind : System.Web.UI.Page
    {
        protected void Page_Load(object sender, EventArgs e)
        {
            Response.Write("Hello World. Time is: " + DateTime.Now.ToString() );
            Response.End();
        }
    }

while the Markup page only contains some static output via an expression:

<%@ Page Language="C#" AutoEventWireup="false" 
         CodeBehind="HelloWorld_Markup.aspx.cs" 
        Inherits="AspNetFrameworksPerformance.HelloWorld_Markup" %>
Hello World. Time is <%= DateTime.Now %>

ASP.NET WebPages

WebPages is the freestanding Razor implementation of ASP.NET. Here's the simple HelloWorld.cshtml page:

Hello World @DateTime.Now

WCF REST

WCF REST was the token REST implementation for ASP.NET before WebAPI and the inbetween step from ASP.NET AJAX. I'd like to forget that this technology was ever considered for production use, but I'll include it here. Here's an OperationContract class:

    [ServiceContract(Namespace = "")]
    [AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]
    public class WcfService
    {

        [OperationContract]
        [WebGet]
        public Stream HelloWorld()
        {
            var data = Encoding.Unicode.GetBytes("Hello World" + DateTime.Now.ToString());
            var ms = new MemoryStream(data);

            // Add your operation implementation here
            return ms;
        }
    }

WCF REST can return arbitrary results by returning a Stream object and a content type. The code above turns the string result into a stream and returns that back to the client.

ASP.NET AJAX (ASMX Services)

I also wanted to test ASP.NET AJAX services because prior to WebAPI this is probably still the most widely used AJAX technology for the ASP.NET stack today.

Unfortunately I was completely unable to get this running on my Windows 8 machine. Visual Studio 2012  removed adding of ASP.NET AJAX services, and when I tried to manually add the service and configure the script handler references it simply did not work - I always got a SOAP response for GET and POST operations. No matter what I tried I always ended up getting XML results even when explicitly adding the ScriptHandler. So, I didn't test this (but the code is there - you might be able to test this on a Windows 7 box).

ASP.NET MVC

Next up is probably the most popular ASP.NET technology at the moment: MVC. Here's the small controller:

    public class MvcPerformanceController : Controller
    {
        public ActionResult Index()
        {
            return View();
        }

        public ActionResult HelloWorldCode()
        {
            return new ContentResult() 
            { Content = "Hello World. Time is: " + DateTime.Now.ToString() };
        }
    }

ASP.NET WebAPI

Next up is WebAPI which looks kind of similar to MVC. Except here I have to use a StringContent result to return the response:

    public class WebApiPerformanceController : ApiController
    {
        [HttpGet]
        public HttpResponseMessage HelloWorldCode()
        {
            return new HttpResponseMessage() 
            { Content = new StringContent("Hello World. Time is: " + DateTime.Now.ToString(), Encoding.UTF8, "text/plain") };
        }
   }

Testing

Take a minute to think about each of the technologies… and take a guess which you think is most efficient in raw throughput. The fastest should be pretty obvious, but the others - maybe not so much.

The testing I did is pretty informal since it was mainly to satisfy my curiosity - here's how I did this: I used Apache Bench (ab.exe) from a full Apache HTTP installation to run and log the test results of hitting the server. ab.exe is a small executable that lets you hit a URL repeatedly and provides counter information about the number of requests, requests per second etc. ab.exe and the batch file are located in the \LoadTests folder of the project.

An ab.exe command line  looks like this:

ab.exe -n100000 -c20 http://localhost/aspnetperf/api/HelloWorld

which hits the specified URL 100,000 times with a load factor of 20 concurrent requests. This results in output like this:

Console[4] 

It's a great way to get a quick and dirty performance summary. Run it a few times to make sure there's not a large amount of varience. You might also want to do an IISRESET to clear the Web Server. Just make sure you do a short test run to warm up the server first - otherwise your first run is likely to be skewed downwards. ab.exe also allows you to specify headers and provide POST data and many other things if you want to get a little more fancy. Here all tests are GET requests to keep it simple.

I ran each test:

  • 100,000 iterations
  • Load factor of 20 concurrent connections
  • IISReset before starting
  • A short warm up run for API and MVC to make sure startup cost is mitigated

Here is the batch file I used for the test:

IISRESET

REM make sure you add 
REM C:\Program Files (x86)\Apache Software Foundation\Apache2.2\bin
REM to your path so ab.exe can be found

REM Warm up
ab.exe -n100 -c20 http://localhost/aspnetperf/MvcPerformance/HelloWorldJson
ab.exe -n100 -c20 http://localhost/aspnetperf/api/HelloWorldJson ab.exe -n100 -c20 http://localhost/AspNetPerf/WcfService.svc/HelloWorld ab.exe -n100000 -c20 http://localhost/aspnetperf/handler.ashx > handler.txt ab.exe -n100000 -c20 http://localhost/aspnetperf/HelloWorld_CodeBehind.aspx > AspxCodeBehind.txt ab.exe -n100000 -c20 http://localhost/aspnetperf/HelloWorld_Markup.aspx > AspxMarkup.txt ab.exe -n100000 -c20 http://localhost/AspNetPerf/WcfService.svc/HelloWorld > Wcf.txt ab.exe -n100000 -c20 http://localhost/aspnetperf/MvcPerformance/HelloWorldCode > Mvc.txt ab.exe -n100000 -c20 http://localhost/aspnetperf/api/HelloWorld > WebApi.txt

I ran each of these tests 3 times and took the average score for Requests/second, with the machine otherwise idle. I did see a bit of variance when running many tests but the values used here are the medians. Part of this has to do with the fact I ran the tests on my local machine - result would probably more consistent running the load test on a separate machine hitting across the network.

I ran these tests locally on my laptop which is a Dell XPS with quad core Sandibridge I7-2720QM @ 2.20ghz and a fast SSD drive on Windows 8. CPU load during tests ran to about 70% max across all 4 cores (IOW, it wasn't overloading the machine). Ideally you can try running these tests on a separate machine hitting the local machine. If I remember correctly IIS 7 and 8 on client OSs don't throttle so the performance here should be representative of what you can expect in a live scenario.

Results

Ok, let's cut straight to the chase. Below are the results from the tests…

Results[4]

The chart shows Requests per second for the different technologies. It's not surprising that the handler was fastest. But it was a bit surprising to me that the next fastest was WebForms and especially Web Forms with markup over a CodeBehind page. WebPages also fared fairly well. MVC and WebAPI are a little slower and the slowest by far is WCF REST (which again I find surprising).

As mentioned at the start the raw throughput tests are not overly practical as they don't test scripting performance for the HTML generation engines or serialization performances of the data engines. All it really does is give you an idea of the raw throughput for the technology from time of request to reaching the endpoint and returning minimal text data back to the client which indicates full round trip performance.

But it's still interesting to see that Web Forms performs better in throughput than either MVC, WebAPI or WebPages. It'd be interesting to try this with a few pages that actually have some parsing logic on it, but that's beyond the scope of this throughput test.

But what's also amazing about this test is the sheer amount of traffic that a laptop computer is handling. Even the slowest tech managed 5700 requests a second, which is one hell of a lot of requests if you extrapolate that out over a 24 hour period. Remember these are not static pages, but dynamic requests that are being served.

Another test - JSON Data Service Results

The second test I used a JSON result from several of the technologies. I didn't bother running WebForms and WebPages through this test since that doesn't make a ton of sense to return data from the them (OTOH, returning text from the APIs didn't make a ton of sense either :-)

In these tests I have a small Person class that gets serialized and then returned to the client. The Person class looks like this:

    public class Person
    {
        public Person()
        {
            Id = 10;
            Name = "Rick";
            Entered = DateTime.Now;
        }

        public int Id { get; set; }
        public string Name { get; set; }
        public DateTime Entered { get; set; }
    }

Here are the updated handler classes that use Person:

Handler

    public class Handler : IHttpHandler
    {
        public void ProcessRequest(HttpContext context)
        {
            var action = context.Request.QueryString["action"];
            if (action == "json")
                JsonRequest(context);
            else
                TextRequest(context);
        }

        public void TextRequest(HttpContext context)
        {
            context.Response.ContentType = "text/plain";
            context.Response.Write("Hello World. Time is: " + DateTime.Now.ToString());
        }

        public void JsonRequest(HttpContext context)
        {
            var json = JsonConvert.SerializeObject(new Person(), Formatting.None);
            context.Response.ContentType = "application/json";
            context.Response.Write(json);
        }

        public bool IsReusable
        {
            get
            {
                return true;
            }
        }
    }

This code adds a little logic to check for a action query string and route the request to an optional JSON result method. To generate JSON, I'm using the same JSON.NET serializer (JsonConvert.SerializeObject) used in Web API to create the JSON response.

WCF REST

 

    [ServiceContract(Namespace = "")]
    [AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]
    public class WcfService
    {

        [OperationContract]
        [WebGet]
        public Stream HelloWorld()
        {
            var data = Encoding.Unicode.GetBytes("Hello World " + DateTime.Now.ToString());
            var ms = new MemoryStream(data);

            // Add your operation implementation here
            return ms;
        }
        
        [OperationContract]
        [WebGet(ResponseFormat=WebMessageFormat.Json,BodyStyle=WebMessageBodyStyle.WrappedRequest)]
        public Person HelloWorldJson()
        {
            // Add your operation implementation here
            return new Person();
        }

    }

For WCF REST all I have to do is add a method with the Person result type.

 

ASP.NET MVC

    public class MvcPerformanceController : Controller
    {
        //
        // GET: /MvcPerformance/

        public ActionResult Index()
        {
            return View();
        }

        public ActionResult HelloWorldCode()
        {
            return new ContentResult() 
            { Content = "Hello World. Time is: " + DateTime.Now.ToString() };
        }

        public JsonResult HelloWorldJson()
        {
            return Json(new Person(), JsonRequestBehavior.AllowGet);
        }

    }

For MVC all I have to do for a JSON response is return a JSON result. ASP.NET internally uses JavaScriptSerializer.

ASP.NET WebAPI

    public class WebApiPerformanceController : ApiController
    {
        [HttpGet]
        public HttpResponseMessage HelloWorldCode()
        {
            return new HttpResponseMessage() 
            { Content = new StringContent("Hello World. Time is: " + DateTime.Now.ToString(), Encoding.UTF8, "text/plain") };
        }

        [HttpGet]
        public Person HelloWorldJson()
        {
            return new Person();
        }

        [HttpGet]
        public HttpResponseMessage HelloWorldJson2()
        {
            var response = new HttpResponseMessage(HttpStatusCode.OK);
            response.Content = new ObjectContent<Person>(new Person(),
                            GlobalConfiguration.Configuration.Formatters.JsonFormatter);
            return response;
        }

   }

Testing and Results

To run these data requests I used the following ab.exe commands:

REM JSON RESPONSES
ab.exe -n100000 -c20 http://localhost/aspnetperf/Handler.ashx?action=json > HandlerJson.txt
ab.exe -n100000 -c20 http://localhost/aspnetperf/MvcPerformance/HelloWorldJson > MvcJson.txt
ab.exe -n100000 -c20 http://localhost/aspnetperf/api/HelloWorldJson > WebApiJson.txt 
ab.exe -n100000 -c20 http://localhost/AspNetPerf/WcfService.svc/HelloWorldJson > WcfJson.txt

The results from this test run are a bit interesting in that the WebAPI test improved performance significantly over returning plain string content.

Here are the results:

Results2[4] 

The performance for each technology drops a little bit except for WebAPI which is up quite a bit! From this test it appears that WebAPI is actually significantly better performing returning a JSON response, rather than a plain string response.

Snag with Apache Benchmark and 'Length Failures'

I ran into a little snag with Apache Benchmark, which was reporting failures for my Web API requests when serializing. As the graph shows performance improved significantly from with JSON results from 5580 to 6530 or so which is a 15% improvement (while all others slowed down by 3-8%).

However, I was skeptical at first because the WebAPI test reports showed a bunch of errors on about 10% of the requests. Check out this report:

ResultsCommand2

Notice the Failed Request count. What the hey? Is WebAPI failing on roughly 10% of requests when sending JSON?

Turns out: No it's not! But it took some sleuthing to figure out why it reports these failures. At first I thought that Web API was failing, and so to make sure I re-ran the test with Fiddler attached and runiisning the ab.exe test by using the -X switch:

ab.exe -n100 -c10 -X localhost:8888 http://localhost/aspnetperf/api/HelloWorldJson

which showed that indeed all requests where returning proper HTTP 200 results with full content.

However ab.exe was reporting the errors. After some closer inspection it turned out that the dates varying in size altered the response length in dynamic output.

For example: these two results:

{"Id":10,"Name":"Rick","Entered":"2012-09-04T10:57:24.841926-10:00"}

{"Id":10,"Name":"Rick","Entered":"2012-09-04T10:57:24.8519262-10:00"}

are different in length for the number which results in 68 and 69 bytes respectively. The same URL produces different result lengths which is what ab.exe reports. I didn't notice at first bit the same is happening when running the ASHX handler with JSON.NET result since it uses the same serializer that varies the milliseconds.

Moral: You can typically ignore Length failures in Apache Benchmark and when in doubt check the actual output with Fiddler. Note that the other failure values are accurate though.

Another interesting Side Note: Perf drops over Time

As I was running these tests repeatedly I was finding that performance steadily dropped from a startup peak to a 10-15% lower stable level. IOW, with Web API I'd start out with around 6500 req/sec and in subsequent runs it keeps dropping until it would stabalize somewhere around 5900 req/sec occasionally jumping lower. For these tests this is why I did the IIS RESET and warm up for individual tests. This is a little puzzling. Looking at Process Monitor while the test are running memory very quickly levels out as do handles and threads, on the first test run. Subsequent runs everything stays stable, but the performance starts going downwards. This applies to all the technologies - Handlers, Web Forms, MVC, Web API - curious to see if others test this and see similar results.

Doing an IISRESET then resets everything and performance starts off at peak again…

Summary

As I stated at the outset, these were informal to satiate my curiosity not to prove that any technology is better or even faster than another. While there clearly are differences in performance the differences (other than WCF REST which was by far the slowest and the raw handler which was by far the highest) are relatively minor, so there is no need to feel that any one technology is a runaway standout in raw performance. Choosing a technology is about more than pure performance but also about the adequateness for the job and the easy of implementation. The strengths of each technology will make for any minor performance difference we see in these tests.

However, to me it's important to get an occasional reality check and compare where new technologies are heading. Often times old stuff that's been optimized and designed for a time of less horse power can utterly blow the doors off newer tech and simple checks like this let you compare. Luckily we're seeing that much of the new stuff performs well even in V1.0 which is great.

To me it was very interesting to see Web API perform relatively badly with plain string content, which originally led me to think that Web API might not be properly optimized just yet. For those that caught my Tweets late last week regarding WebAPI's slow responses was with String content which is in fact considerably slower. Luckily where it counts with serialized JSON and XML WebAPI actually performs better. But I do wonder what would make generic string content slower than serialized code?

This stresses another point: Don't take a single test as the final gospel and don't extrapolate out from a single set of tests. Certainly Twitter can make you feel like a fool when you post something immediate that hasn't been fleshed out a little more <blush>. Egg on my face. As a result I ended up screwing around with this for a few hours today to compare different scenarios. Well worth the time…

I hope you found this useful, if not for the results, maybe for the process of quickly testing a few requests for performance and charting out a comparison. Now onwards with more serious stuff…

Resources

© Rick Strahl, West Wind Technologies, 2005-2012
Posted in ASP.NET  Web Api  

Passing multiple simple POST Values to ASP.NET Web API

$
0
0

A few weeks backs  I wrote about what does and doesn't work With POST values in ASP.NET Web API when it comes to sending data to a Web API controller. One of the features that doesn't work out of the box - somewhat unexpectedly -  is the ability to map POST form variables to simple parameters of a Web API method.

For example imagine you have this form and you want to post this data to a Web API end point like this via AJAX:

    <form>
        Name: <input type="name" name="name" value="Rick" />
        Value: <input type="value" name="value" value="12" />
        Entered: <input type="entered" name="entered" value="12/01/2011" />
        <input type="button" id="btnSend" value="Send" />
    </form>

    <script type="text/javascript">
    $("#btnSend").click( function() {                
        $.post("samples/PostMultipleSimpleValues?action=kazam",
               $("form").serialize(),
               function (result) {
                   alert(result);                   
               });
    });
    </script>

or you might do this more explicitly by creating a simple client map and specifying the POST values directly by hand:

$.post("samples/PostMultipleSimpleValues?action=kazam",
        { name: "Rick", value: 1, entered: "12/01/2012" },
        function (result) {
            alert(result);                   
        });

On the wire this generates a simple POST request with Url Encoded values in the content:

POST /AspNetWebApi/samples/PostMultipleSimpleValues?action=kazam HTTP/1.1
Host: localhost
User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:15.0) Gecko/20100101 Firefox/15.0.1
Accept: application/json
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
X-Requested-With: XMLHttpRequest
Referer: http://localhost/AspNetWebApi/FormPostTest.html
Content-Length: 41
Pragma: no-cache
Cache-Control: no-cache
name=Rick&value=12&entered=12%2F10%2F2011

Seems simple enough, right? We are basically posting 3 form variables and 1 query string value to the server.

Unfortunately Web API can't handle this request out of the box. If I create a method like this:

[HttpPost]
public string PostMultipleSimpleValues(string name, int value, DateTime entered, string action = null)
{
    return string.Format("Name: {0}, Value: {1}, Date: {2}, Action: {3}", name, value, entered, action);
}
You'll find that you get an HTTP 404 error and
{
  "Message": "No HTTP resource was found that matches the request URI…"
}

Yes, it's possible to pass multiple POST parameters of course, but Web API expects you to use Model Binding for this - mapping the post parameters to a strongly typed .NET object, not to single parameters. Alternately you can also accept a FormDataCollection parameter on your API method to get a name value collection of all POSTed values. If you're using JSON only, using the dynamic JObject/JValue objects might also work.

Model Binding is fine in many use cases, but can quickly become overkill if you only need to pass a couple of simple parameters to many methods. Especially in applications with many, many AJAX callbacks the 'parameter mapping type' per method signature can lead to serious class pollution in a project very quickly. Simple POST variables are also commonly used in AJAX applications to pass data to the server, even in many complex public APIs. So this is not an uncommon use case, and - maybe more so a behavior that I would have expected Web API to support natively. The question "Why aren't my POST parameters mapping to Web API method parameters" is already a frequent one…

So this is something that I think is fairly important, but unfortunately missing in the base Web API installation.

Creating a Custom Parameter Binder

Luckily Web API is greatly extensible and there's a way to create a custom Parameter Binding to provide this functionality! Although this solution took me a long while to find and then only with the help of some folks at Microsoft (thanks Hong Mei!!!), it's not difficult to hook up in your own projects once you know what to implement. It requires one small class and a GlobalConfiguration hookup.

Web API parameter bindings allow you to intercept processing of individual parameters - they deal with mapping parameters to the signature as well as converting the parameters to the actual values that are returned.

Here's the implementation of the SimplePostVariableParameterBinding class:

(updated 9/24/2012: Fixed bug with non-form data trying to read query string values)

public class SimplePostVariableParameterBinding : HttpParameterBinding
{
    private const string MultipleBodyParameters = "MultipleBodyParameters";

    public SimplePostVariableParameterBinding(HttpParameterDescriptor descriptor)
        : base(descriptor)
    {
    }

    /// <summary>
    /// Check for simple binding parameters in POST data. Bind POST
    /// data as well as query string data
    /// </summary>
    /// <param name="metadataProvider"></param>
    /// <param name="actionContext"></param>
    /// <param name="cancellationToken"></param>
    /// <returns></returns>
    public override Task ExecuteBindingAsync(ModelMetadataProvider metadataProvider,
                                                HttpActionContext actionContext,
                                                CancellationToken cancellationToken)
    {               
        string stringValue = null;

        NameValueCollection col = TryReadBody(actionContext.Request);
        if (col != null)
            stringValue = col[Descriptor.ParameterName];

        // try reading query string if we have no POST/PUT match
        if (stringValue == null)
        {
            var query = actionContext.Request.GetQueryNameValuePairs();
            if (query != null)
            {
                var matches = query.Where(kv => kv.Key.ToLower() == Descriptor.ParameterName.ToLower());
                if (matches.Count() > 0)
                    stringValue = matches.First().Value;
            }
        }

        object value = StringToType(stringValue);

        // Set the binding result here
        SetValue(actionContext, value);

        // now, we can return a completed task with no result
        TaskCompletionSource<AsyncVoid> tcs = new TaskCompletionSource<AsyncVoid>();
        tcs.SetResult(default(AsyncVoid));
        return tcs.Task;
    }


    /// <summary>
    /// Method that implements parameter binding hookup to the global configuration object's
    /// ParameterBindingRules collection delegate.
    /// 
    /// This routine filters based on POST/PUT method status and simple parameter
    /// types.
    /// </summary>
    /// <example>
    /// GlobalConfiguration.Configuration.
    ///       .ParameterBindingRules
    ///       .Insert(0,SimplePostVariableParameterBinding.HookupParameterBinding);
    /// </example>    
    /// <param name="descriptor"></param>
    /// <returns></returns>
    public static HttpParameterBinding HookupParameterBinding(HttpParameterDescriptor descriptor)
    {
        var supportedMethods = descriptor.ActionDescriptor.SupportedHttpMethods;      

        // Only apply this binder on POST and PUT operations
        if (supportedMethods.Contains(HttpMethod.Post) ||
            supportedMethods.Contains(HttpMethod.Put))
        {
            var supportedTypes = new Type[] { typeof(string), 
                                                typeof(int), 
                                                typeof(decimal), 
                                                typeof(double), 
                                                typeof(bool),
                                                typeof(DateTime),
                                                typeof(byte[])
                                            };            

            if (supportedTypes.Where(typ => typ == descriptor.ParameterType).Count() > 0)
                return new SimplePostVariableParameterBinding(descriptor);
        }

        return null;
    }


    private object StringToType(string stringValue)
    {
        object value = null;

        if (stringValue == null)
            value = null;
        else if (Descriptor.ParameterType == typeof(string))
            value = stringValue;
        else if (Descriptor.ParameterType == typeof(int))
            value = int.Parse(stringValue, CultureInfo.CurrentCulture);
        else if (Descriptor.ParameterType == typeof(Int32))
            value = Int32.Parse(stringValue, CultureInfo.CurrentCulture);
        else if (Descriptor.ParameterType == typeof(Int64))
            value = Int64.Parse(stringValue, CultureInfo.CurrentCulture);
        else if (Descriptor.ParameterType == typeof(decimal))
            value = decimal.Parse(stringValue, CultureInfo.CurrentCulture);
        else if (Descriptor.ParameterType == typeof(double))
            value = double.Parse(stringValue, CultureInfo.CurrentCulture);
        else if (Descriptor.ParameterType == typeof(DateTime))
            value = DateTime.Parse(stringValue, CultureInfo.CurrentCulture);
        else if (Descriptor.ParameterType == typeof(bool))
        {
            value = false;
            if (stringValue == "true" || stringValue == "on" || stringValue == "1")
                value = true;
        }
        else
            value = stringValue;

        return value;
    }

    /// <summary>
    /// Read and cache the request body
    /// </summary>
    /// <param name="request"></param>
    /// <returns></returns>
    private NameValueCollection TryReadBody(HttpRequestMessage request)
    {
       object result = null;
        
        // try to read out of cache first
        if (!request.Properties.TryGetValue(MultipleBodyParameters, out result))
        {
            var contentType = request.Content.Headers.ContentType;

            // only read if there's content and it's form data
            if (contentType == null || contentType.MediaType != "application/x-www-form-urlencoded")
            {
                // Nope no data
                result = null;
            }
            else
            {
                // parsing the string like firstname=Hongmei&lastname=ASDASD            
                result = request.Content.ReadAsFormDataAsync().Result;
            }

            request.Properties.Add(MultipleBodyParameters, result);            
        }

        return result as NameValueCollection;
    }

    private struct AsyncVoid
    {
    }
}

 

The ExecuteBindingAsync method is fired for each parameter that is mapped and sent for conversion. This custom binding is fired only if the incoming parameter is a simple type (that gets defined later when I hook up the binding), so this binding never fires on complex types or if the first type is not a simple type.

For the first parameter of a request the Binding first reads the request body into a NameValueCollection and caches that in the request.Properties collection. The request body can only be read once, so the first parameter request reads it and then caches it. Subsequent parameters then use the cached POST value collection. Once the form collection is available the value of the parameter is read, and the value is translated into the target type requested by the Descriptor. SetValue writes out the value to be mapped.

Once you have the ParameterBinding in place, the binding has to be assigned. This is done along with all other Web API configuration tasks at application startup in global.asax's Application_Start:

// Attach simple post variable binding
GlobalConfiguration.Configuration.
        .ParameterBindingRules
        .Insert(0,SimplePostVariableParameterBinding.HookupParameterBinding);

The hookup code calls the static HookupParameterBinding method of the SimplePostVariableParameterBinding class that provides the delegate that the Insert method requires. The delegate's job is to check which type of requests the ParameterBinding should handle. The logic in HookupParameterBinding checks whether the request is POST or PUT and whether the parameter type is one of the simple types that is supported. Web API calls this delegate once for each method signature it tries to map and the function  returns null to indicate it's not handling this parameter, or it returns a new parameter binding instance - in this case the SimplePostVariableParameterBinding.

As a consumer of this class the one line above is all you need!

Once the parameter binding and this hook up code is in place, you can now pass simple POST values to methods with simple parameters. The examples I showed above should now work in addition to the standard bindings.

Summary

Clearly this is not easy to discover. I spent quite a bit of time digging through the Web API source trying to figure this out on my own without much luck. It took Hong Mei from Microsoft to provide a base example, so I can't take credit for this solution :-). But once you know where to look, Web API is brilliantly extensible to make it relatively easy to customize the parameter behavior.

I'm very stoked that this got resolved  - in the last two months I've had two customers with projects that decided not to use Web API in AJAX heavy SPA applications because this POST variable mapping wasn't available. This custom parameter binding in Web API might actually change their mind to still switch back and take advantage of the many great features in Web API. I too frequently use plain POST variables for communicating with server AJAX handlers and while I work around this (with untyped JObject or the Form collections mostly), I prefer in many cases to push simple POST variables to the server - it's simply more convenient and more logical in many situations.

I said this in my last post on POST data and say it again here:

I think POST to method parameter mapping should have been shipped in the box with Web API! Without knowing about this limitation and seeing that query string values match, the expectation is that simple POST variables should also map to parameters. I hope Microsoft considers including this type of functionality natively in the next version of Web API natively or at least as a built-in HttpParameterBinding that can be just added. Especially given that this binding doesn't adversely affect existing bindings or even their performance.

Resources

© Rick Strahl, West Wind Technologies, 2005-2012
Posted in Web Api  AJAX  

Creating STA COM compatible ASP.NET Applications

$
0
0

When building ASP.NET applications that interface with old school COM objects like those created with VB6 or Visual FoxPro (MTDLL), it's extremely important that the threads that are serving requests use Single Threaded Apartment Threading. STA is a COM built-in technology that allows essentially single threaded components to operate reliably in a multi-threaded environment. STA's guarantee that COM objects instantiated on a specific thread stay on that specific thread and any access to a COM object from another thread automatically marshals that thread to the STA thread. The end effect is that you can have multiple threads, but a COM object instance lives on a fixed never changing thread.

ASP.NET by default uses MTA (multi-threaded apartment) threads which are truly free spinning threads that pay no heed to COM object marshaling. This is vastly more efficient than STA threading which has a bit of overhead in determining whether it's OK to run code on a given thread or whether some sort of thread/COM marshaling needs to occur. MTA COM components can be very efficient, but STA COM components in a multi-threaded environment always tend to have a fair amount of overhead.

It's amazing how much COM Interop I still see today so while it seems really old school to be talking about this topic, it's actually quite apropos for me as I have many customers using legacy COM systems that need to interface with other .NET applications. In this post I'm consolidating some of the hacks I've used to integrate with various ASP.NET technologies when using STA COM Components.

STA in ASP.NET

Support for STA threading in the ASP.NET framework is fairly limited. Specifically only the original ASP.NET WebForms technology supports STA threading directly via its STA Page Handler implementation or what you might know as ASPCOMPAT mode. For WebForms running STA components is as easy as specifying the ASPCOMPAT attribute in the @Page tag:

<%@ Page Language="C#" AspCompat="true" %>

which runs the page in STA mode. Removing it runs in MTA mode. Simple.

Unfortunately all other ASP.NET technologies built on top of the core ASP.NET engine do not support STA natively. So if you want to use STA COM components in MVC or with class ASMX Web Services, there's no automatic way like the ASPCOMPAT keyword available.

So what happens when you run an STA COM component in an MTA application? In low volume environments - nothing much will happen. The COM objects will appear to work just fine as there are no simultaneous thread interactions and the COM component will happily run on a single thread or multiple single threads one at a time. So for testing running components in MTA environments may appear to work just fine.

However as load increases and threads get re-used by ASP.NET COM objects will end up getting created on multiple different threads. This can result in crashes or hangs, or data corruption in the STA components which store their state in thread local storage on the STA thread. If threads overlap this global store can easily get corrupted which in turn causes problems. STA ensures that any COM object instance loaded always stays on the same thread it was instantiated on.

What about COM+?

COM+ is supposed to address the problem of STA in MTA applications by providing an abstraction with it's own thread pool manager for COM objects. It steps in to the COM instantiation pipeline and hands out COM instances from its own internally maintained STA Thread pool. This guarantees that the COM instantiation threads are STA threads if using STA components.

COM+ works, but in my experience the technology is very, very slow for STA components. It adds a ton of overhead and reduces COM performance noticably in load tests in IIS. COM+ can make sense in some situations but for Web apps with STA components it falls short. In addition there's also the need to ensure that COM+ is set up and configured on the target machine and the fact that components have to be registered in COM+. COM+ also keeps components up at all times, so if a component needs to be replaced the COM+ package needs to be unloaded (same is true for IIS hosted components but it's more common to manage that).

COM+ is an option for well established components, but native STA support tends to provide better performance and more consistent usability, IMHO.

What about HttpRuntime's apartmentThreading Key?

You can specify the following in Web.Config:

<system.web>
<httpRuntime apartmentThreading="true"/> </system.web>

Looks promising right? Unfortunately it appears that this only works as a global flag for WebForms and is equivalent of specifying ASPCOMPAT in all WebForms pages. It has no effect on MVC, Web Services, WCF, Handlers etc. unfortunately.

STA for non supporting ASP.NET Technologies

As mentioned above only WebForms supports STA natively. However, by utilizing the WebForms ASP.NET Page handler internally it's actually possible to trick various other ASP.NET technologies and let them work with STA components. This is ugly but I've used each of these in various applications and I've had minimal problems making them work with FoxPro STA COM components which is about as dififcult as it gets for COM Interop in .NET.

In this post I summarize several STA workarounds that enable you to use STA threading with these ASP.NET Technologies:

  • ASP.NET HttpHandlers
  • ASMX Web Services
  • ASP.NET MVC
  • WCF Web Services
  • ASP.NET Web API

ASP.NET HttpHandlers

HttpHandlers are pretty low to the metal in ASP.NET: They are the lowest level hookable endpoint in ASP.NET for writing application level code. Handlers tend to be for framework level components that build higher level functionality on top, but they can also be used for the highest performance operations that don't need the overhead of higher level frameworks like MVC, WebForms, WebAPI etc. In pure throughput HttpHandlers are as fast as it gets for ASP.NET requests.

As such I'm not sure how useful this is for STA components since by definition STA COM components have a fair bit of overhead when called compared to native .NET code so that diminshes much of the performance benefit of handlers in the first place.

Nevertheless, the base STA Handler implementation demonstrates the core trick that is used to allow STA capable ASP.NET code. The trick to this is the ASP.NET WebForms System.Web.UI.Page class which is an HttpHandler implementation and which also supports an ASPCOMPAT mode that provides STA threads to WebForms pages. As it turns out the STA functionality is wired into the Page class itself via the AspCompatBeginRequest() and AspCompatEndRequest() methods which can be fired from any HttpHandler implementation.

To create an Sta capable handler implementation the following code can be used:

using System;
using System.Web;
using System.Web.SessionState;

namespace FoxProAspNet
{
    /// <summary>
    /// This handler allows SOAP Web Services to run in STA mode
    /// 
    /// Note: this will NOT work with ASP.NET AJAX services as
    /// these services bypass the handler mapping and perform
    /// all work in a module (RestModule).
    /// </summary>
    public abstract class StaHttpHandler :
        System.Web.UI.Page, IHttpAsyncHandler
    {
        protected override void OnInit(EventArgs e)
        {
            this.ProcessRequestSta(this.Context);

            // immediately stop the request after we're done processing
            this.Context.ApplicationInstance.CompleteRequest();
        }

        /// <summary>
        /// This actually should never fire but has to be here for IHttpHandler
        /// </summary>
        /// <param name="context"></param>
        public void ProcessRequest(HttpContext context)
        {
            // internally runs async request so Begin/EndProcessRequest are called
            base.ProcessRequest(context);
        }

        /// <summary>
        /// This method should receive the real processing code
        /// </summary>
        /// <param name="context"></param>
        public abstract void ProcessRequestSta(HttpContext context);


        /// <summary>
        /// This is what handles the STA thread creation/pool thread queing
        /// </summary>
        /// <param name="context"></param>
        /// <param name="cb"></param>
        /// <param name="extraData"></param>
        /// <returns></returns>
        public IAsyncResult BeginProcessRequest(
            HttpContext context, AsyncCallback cb, object extraData)
        {
            return this.AspCompatBeginProcessRequest(context, cb, extraData);
        }


        public void EndProcessRequest(IAsyncResult result)
        {
            this.AspCompatEndProcessRequest(result);
        }
    }

    /// <summary>
    /// Subclass this if your handler needs session state
    /// </summary>
    public class StaHttpHandlerWithSessionState :
                    StaHttpHandler, IRequiresSessionState
    {
        public override void ProcessRequestSta(HttpContext context)
        {
        }

    }

}

To implement the class simply inherit from StaHandler:

public class StaHandler : StaHttpHandler
{
    public void ProcessRequest(HttpContext context)
    {
        ProcessRequestSta(context);
    }

    public override void ProcessRequestSta(HttpContext context)
    {
        context.Response.ContentType = "text/plain";
        context.Response.Write("Hello World " + Thread.CurrentThread.GetApartmentState());
    }     
}

The StaHandler implementation basically overrides the BeginProcessRequest and EndProcessRequest methods and routes the AspCompatBeginRequest/EndRequest methods. By doing so the request gets an STA thread. BeginProcessRequest implicitly fires the OnInit where we can then start executing STA safe code. My base class simply routes to an alternate ProcessRequestSta method which can be implemented in an inherited class.

Originally I thought we should just be able to fire this.ProcessRequest() directly, but it turns out that this causes OnInit() to be fired again, resulting in an eventual StackOverflow. So in order to make this work a method with a different name is required.

ASMX Web Services

ASMX Web Services can use basically the same trick used above, by creating a custom handler class and then instantiating the WebService handler from it and firing it from the Init. WebService itself isn't the handler so the process of getting the handler requires invoking the WebService - actually the ScriptService (that also supports ASP.NET AJAX Services) -HandlerFactory and then calling its ProcessRequest method.

Here's the code for this process:

(updated to ensure that ASP.NET AJAX Services can still run. Note ASP.NET AJAX is not supported for STA operation)

namespace FoxProAspNet
{
    /// <summary>
    /// This handler allows SOAP Web Services to run in STA mode
    /// 
    /// Note: this will NOT work with ASP.NET AJAX services as
    /// these services bypass the handler mapping and perform
    /// all work in a module (RestModule).
    /// </summary>
    public class WebServiceStaHandler :
        System.Web.UI.Page, IHttpAsyncHandler
    {
        /// <summary>
        /// Cache the HttpHandlerFactory lookup
        /// </summary>
        static IHttpHandlerFactory _HttpHandlerFactory = null;
        static object _HandlerFactoryLock = new object();

        protected override void OnInit(EventArgs e)
        {
            var factory = this.GetScriptHandlerFactory();

            IHttpHandler handler =
                factory.GetHandler(
                    this.Context,
                    this.Context.Request.HttpMethod,
                    this.Context.Request.FilePath,
                    this.Context.Request.PhysicalPath);
            handler.ProcessRequest(this.Context);

            // immediately stop the request after we're done processing
            this.Context.ApplicationInstance.CompleteRequest();
        }


        /// <summary>
        /// Loader for the script handler factory
        /// </summary>
        IHttpHandlerFactory GetScriptHandlerFactory()
        {
            if (_HttpHandlerFactory != null)
                return _HttpHandlerFactory;

            IHttpHandlerFactory factory = null;

            lock (_HandlerFactoryLock)
            {
                if (_HttpHandlerFactory != null)
                    return _HttpHandlerFactory;

                try
                {
                    // Try to load the ScriptServiceHandlerFactory
                    // class is internal requires reflection so this requires full trust
                    var assembly = typeof(JavaScriptSerializer).Assembly;
                    var shf = assembly.GetTypes().Where(t => t.Name == "ScriptHandlerFactory").FirstOrDefault();
                    factory = Activator.CreateInstance(shf) as IHttpHandlerFactory;
                }
                catch { }

                // Fallback to just WebService Handler Factory
                if (factory == null)
                    factory = new WebServiceHandlerFactory();

                _HttpHandlerFactory = factory;
            }

            return factory;
        }


        public override void ProcessRequest(HttpContext context)
        {
            base.ProcessRequest(context);
        }

        public IAsyncResult BeginProcessRequest(
            HttpContext context, AsyncCallback cb, object extraData)
        {
            return this.AspCompatBeginProcessRequest(context, cb, extraData);
        }


        public void EndProcessRequest(IAsyncResult result)
        {
            this.AspCompatEndProcessRequest(result);
        }
    }

    public class AspCompatWebServiceStaHandlerWithSessionState :
                    WebServiceStaHandler, IRequiresSessionState
    {
    }

}

This class overrides the ASP.NET WebForms Page class which has a little known AspCompatBeginProcessRequest() and AspCompatEndProcessRequest() method that is responsible for providing the WebForms ASPCOMPAT functionality. These methods handle routing requests to STA threads. Note there are two classes - one that includes session state and one that does not. If you plan on using ASP.NET Session state use the latter class, otherwise stick to the former. This maps to the EnableSessionState page setting in WebForms. '

This class simply hooks into this functionality by overriding the BeginProcessRequest and EndProcessRequest methods and always forcing it into the AspCompat methods.

The way this works is that BeginProcessRequest() fires first to set up the threads and starts intializing the handler. As part of that process the OnInit() method is fired which is now already running on an STA thread. The code then creates an instance of the actual WebService handler factory and calls its ProcessRequest method to start executing which generates the Web Service result. Immediately after ProcessRequest the request is stopped with Application.CompletRequest() which ensures that the rest of the Page handler logic doesn't fire. This means that even though the fairly heavy Page class is overridden here, it doesn't end up executing any of its internal processing which makes this code fairly efficient.

In a nutshell, we're highjacking the Page HttpHandler and forcing it to process the WebService process handler in the context of the AspCompat handler behavior.

ASP.NET AJAX Web Services are not supported

Note that there's a bit of logic there that loads up the right HttpHandlerFactory. The ASMX extension actually manages both SOAP services and ASP.NET AJAX services. Unfortunately ASP.NET AJAX Services do not work using this STA routing because the actual processing for AJAX requests is actually handled in a module (ScriptModule class) and so requests actually never reach the handler. However, the logic to hook up the ScriptServiceFactory is still required if you use ASP.NET AJAX code in your application - without it the AJAX services fail to run even though the processing has already occurred (what kind of funky design is that?). The code is there merely to ensure that ASP.NET AJAX services can co-exist with STA SOAP Web Services.

Hooking up the Handler

Because the above is an HttpHandler implementation you need to hook up the custom handler and replace the standard ASMX handler. To do this you need to modify the web.config file (here for IIS 7 and IIS Express):

 <configuration>
   <system.webServer>
    <handlers>
      <remove name="WebServiceHandlerFactory-Integrated-4.0" />      
      <add name="Asmx STA Web Service Handler" path="*.asmx" verb="*" 
type="FoxProAspNet.WebServiceStaHandler" precondition="integrated"/> </handlers> </system.webServer> </configuration>

(Note: The name for the WebServiceHandlerFactory-Integrated-4.0 might be slightly different depending on your server version. Check the IIS Handler configuration in the IIS Management Console for the exact name or simply remove the handler from the list there which will propagate to your web.config).

For IIS 5 & 6 (Windows XP/2003) or the Visual Studio Web Server use:

<configuration>
<system.web>
<httpHandlers> <remove path="*.asmx" verb="*" /> <add path="*.asmx" verb="*" type="FoxProAspNet.WebServiceStaHandler" /> </httpHandlers> </system.web>
</configuration>

To test, create a new ASMX Web Service and create a method like this:

    [WebService(Namespace = "http://foxaspnet.org/")]
    [WebServiceBinding(ConformsTo = WsiProfiles.BasicProfile1_1)]
    public class FoxWebService : System.Web.Services.WebService
    {

        [WebMethod]
        public string HelloWorld()
        {
            return "Hello World. Threading mode is: " + 
System.Threading.
Thread.CurrentThread.GetApartmentState(); } }

Run this before you put in the web.config configuration changes and you should get:

Hello World. Threading mode is: MTA

Then put the handler mapping into Web.config and you should see:

Hello World. Threading mode is: STA

And you're on your way to using STA COM components. It's a hack but it works well! I've used this with several high volume Web Service installations with various customers and it's been fast and reliable.

ASP.NET MVC

ASP.NET MVC has quickly become the most popular ASP.NET technology, replacing WebForms for creating HTML output. MVC is more complex to get started with, but once you understand the basic structure of how requests flow through the MVC pipeline it's easy to use and amazingly flexible in manipulating HTML requests. In addition, MVC has great support for non-HTML output sources like JSON and XML, making it an excellent choice for AJAX requests without any additional tools.

Unlike WebForms ASP.NET MVC doesn't support STA threads natively and so some trickery is needed to make it work with STA threads as well.

MVC gets its handler implementation through custom route handlers using ASP.NET's built in routing semantics. To work in an STA handler requires working in the Page Handler as part of the Route Handler implementation.

As with the Web Service handler the first step is to create a custom HttpHandler that can instantiate an MVC request pipeline properly:

public class MvcStaThreadHttpAsyncHandler : Page, IHttpAsyncHandler, IRequiresSessionState
{
    RequestContext reqContext;

    public MvcStaThreadHttpAsyncHandler(RequestContext requestContext)
    {
        if (requestContext == null)
            throw new ArgumentNullException("requestContext");

        reqContext = requestContext;
    }

    public IAsyncResult BeginProcessRequest(HttpContext context, AsyncCallback cb, object extraData)
    {
        return this.AspCompatBeginProcessRequest(context, cb, extraData);
    }

    protected override void OnInit(EventArgs e)
    {
        var controllerName = reqContext.RouteData.GetRequiredString("controller");
        if (string.IsNullOrEmpty(controllerName))
            throw new InvalidOperationException("Could not find controller to execute");

        var controllerFactory = ControllerBuilder.Current.GetControllerFactory();

        IController controller = null;
        try
        {
            controller = controllerFactory.CreateController(reqContext, controllerName);
            if (controller == null)
                throw new InvalidOperationException("Could not find controller: " + controllerName);

        }
        catch
        {
            throw new InvalidOperationException("Could not find controller: " + controllerName +
                                                ". Could be caused by missing default document in virtual.");
        }

        try
        {
            controller.Execute(reqContext);
        }
        finally
        {
            controllerFactory.ReleaseController(controller);
        }

        this.Context.ApplicationInstance.CompleteRequest();
    }

    public void EndProcessRequest(IAsyncResult result)
    {
        this.AspCompatEndProcessRequest(result);
    }

    public override void ProcessRequest(HttpContext httpContext)
    {
        throw new NotSupportedException();
    }
}

This handler code figures out which controller to load and then executes the controller. MVC internally provides the information needed to route to the appropriate method and pass the right parameters. Like the Web Service handler the logic occurs in the OnInit() and performs all the processing in that part of the request.

Next, we need a RouteHandler that can actually pick up this handler. Unlike the Web Service handler where we simply registered the handler, MVC requires a RouteHandler to pick up the handler. RouteHandlers look at the URL's path and based on that decide on what handler to invoke.

The route handler is pretty simple - all it does is load our custom handler:

    public class MvcStaThreadRouteHandler : IRouteHandler
    {
        public IHttpHandler GetHttpHandler(RequestContext requestContext)
        {
            if (requestContext == null)
                throw new ArgumentNullException("requestContext");

            return new MvcStaThreadHttpAsyncHandler(requestContext);
        }
    }

At this point you can instantiate this route handler and force STA requests to MVC by specifying a route. The following sets up the ASP.NET Default Route:

Route mvcRoute = new Route("{controller}/{action}/{id}",
                            new RouteValueDictionary( new { 
                                    controller = "Home", 
                                    action = "Index", 
                                    id = UrlParameter.Optional 
                                }),
                            new MvcStaThreadRouteHandler());
RouteTable.Routes.Add(mvcRoute);

 

To make this code a little easier to work with and mimic the behavior of the routes.MapRoute() functionality extension method that MVC provides, here is an extension method for MapMvcStaRoute():

    public static class RouteCollectionExtensions
    {

        public static void MapMvcStaRoute(this RouteCollection routeTable,
                                     string name,
                                     string url,
                                     object defaults = null)
        {
            Route mvcRoute = new Route(url,
                                       new RouteValueDictionary(defaults),
                                       new MvcStaThreadRouteHandler());
            RouteTable.Routes.Add(mvcRoute);
        }
    }

With this the syntax to add  route becomes a little easier and matches the MapRoute() method:

RouteTable.Routes.MapMvcStaRoute(
    name: "Default",
    url: "{controller}/{action}/{id}",
    defaults: new { controller = "Home", action = "Index", id = UrlParameter.Optional }
);

The nice thing about this route handler, STA Handler and extension method is that it's fully self contained. You can put all three into a single class file and stick it into your Web app, and then simply call MapMvcStaRoute() and it just works. Easy!

To see whether this works create an MVC controller like this:

    public class ThreadTestController : Controller
    {        
        public string ThreadingMode()
        {
            return Thread.CurrentThread.GetApartmentState().ToString();
        }
    }

Try this test both with only the MapRoute() hookup in the RouteConfiguration in which case you should get MTA as the value. Then change the MapRoute() call to MapMvcStaRoute() leaving all the parameters the same and re-run the request. You now should see STA as the result.

You're on your way using STA COM components reliably in ASP.NET MVC.

WCF Web Services running through IIS

WCF Web Services provide a more robust and wider range of services for Web Services. You can use WCF over HTTP, TCP, and Pipes, and WCF services support WS* secure services. There are many features in WCF that go way beyond what ASMX can do. But it's also a bit more complex than ASMX. As a basic rule if you need to serve straight SOAP Services over HTTP I 'd recommend sticking with the simpler ASMX services especially if COM is involved. If you need WS* support or want to serve data over non-HTTP protocols then WCF makes more sense.

WCF is not my forte but I found a solution from Scott Seely on his blog that describes the progress and that seems to work well. I'm copying his code below so this STA information is all in one place and quickly explain.

Scott's code basically works by creating a custom OperationBehavior which can be specified via an [STAOperation] attribute on every method. Using his attribute you end up with a class (or Interface if you separate the contract and class) that looks like this:

    [ServiceContract]
    public class WcfService
    {
        [OperationContract]
        public string HelloWorldMta()
        {
            return Thread.CurrentThread.GetApartmentState().ToString();
        }

        // Make sure you use this custom STAOperationBehavior
        // attribute to force STA operation of service methods
        [STAOperationBehavior]
        [OperationContract]
        public string HelloWorldSta()
        {
            return Thread.CurrentThread.GetApartmentState().ToString();
        }

    } 

Pretty straight forward. The latter method returns STA while the former returns MTA. To make STA work every method needs to be marked up.

The implementation consists of the attribute and OperationInvoker implementation. Here are the two classes required to make this work from Scott's post:

public class STAOperationBehaviorAttribute : Attribute, IOperationBehavior
{
    public void AddBindingParameters(OperationDescription operationDescription,
        System.ServiceModel.Channels.BindingParameterCollection bindingParameters)
    {
    }

    public void ApplyClientBehavior(OperationDescription operationDescription,
        System.ServiceModel.Dispatcher.ClientOperation clientOperation)
    {
        // If this is applied on the client, well, it just doesn’t make sense.
        // Don’t throw in case this attribute was applied on the contract
        // instead of the implementation.
    }

    public void ApplyDispatchBehavior(OperationDescription operationDescription,
        System.ServiceModel.Dispatcher.DispatchOperation dispatchOperation)
    {
        // Change the IOperationInvoker for this operation.
        dispatchOperation.Invoker = new STAOperationInvoker(dispatchOperation.Invoker);
    }

    public void Validate(OperationDescription operationDescription)
    {
        if (operationDescription.SyncMethod == null)
        {
            throw new InvalidOperationException("The STAOperationBehaviorAttribute " +
                "only works for synchronous method invocations.");
        }
    }
}

public class STAOperationInvoker : IOperationInvoker
{
    IOperationInvoker _innerInvoker;
    public STAOperationInvoker(IOperationInvoker invoker)
    {
        _innerInvoker = invoker;
    }

    public object[] AllocateInputs()
    {
        return _innerInvoker.AllocateInputs();
    }

    public object Invoke(object instance, object[] inputs, out object[] outputs)
    {
        // Create a new, STA thread
        object[] staOutputs = null;
        object retval = null;
        Thread thread = new Thread(
            delegate()
            {
                retval = _innerInvoker.Invoke(instance, inputs, out staOutputs);
            });
        thread.SetApartmentState(ApartmentState.STA);
        thread.Start();
        thread.Join();
        outputs = staOutputs;
        return retval;
    }

    public IAsyncResult InvokeBegin(object instance, object[] inputs,
        AsyncCallback callback, object state)
    {
        // We don’t handle async…
        throw new NotImplementedException();
    }

    public object InvokeEnd(object instance, out object[] outputs, IAsyncResult result)
    {
        // We don’t handle async…
        throw new NotImplementedException();
    }

    public bool IsSynchronous
    {
        get { return true; }
    }
}

The key in this setup is the Invoker and the Invoke method which creates a new thread and then fires the request on this new thread.

Because this approach creates a new thread for every request it's not super efficient. There's a bunch of overhead involved in creating the thread and throwing it away after each thread, but it'll work for low volume requests and insure each thread runs in STA mode. If better performance is required it would be useful to create a custom thread manager that can pool a number of STA threads and hand off threads as needed rather than creating new threads on every request.

If your Web Service needs are simple and you need only to serve standard SOAP 1.x requests, I would recommend sticking with ASMX services. It's easier to set up and work with and for STA component use it'll be significantly better performing since ASP.NET manages the STA thread pool for you rather than firing new threads for each request. One nice thing about Scotts code is though that it works in any WCF environment including self hosting. It has no dependency on ASP.NET or WebForms for that matter.

STA - If you must

STA components are a  pain in the ass and thankfully there isn't too much stuff out there anymore that requires it. But when you need it and you need to access STA functionality from .NET at least there are a few options available to make it happen. Each of these solutions is a bit hacky, but they work - I've used all of them in production with good results with FoxPro components.

I hope compiling all of these in one place here makes it STA consumption a little bit easier. I feel your pain :-)

Resources

© Rick Strahl, West Wind Technologies, 2005-2012
Posted in FoxPro   ASP.NET  .NET  COM  

A tiny Utility to recycle an IIS Application Pool

$
0
0

In the last few weeks I've annoyingly been having problems with an area on my Web site. It's basically ancient articles that are using ASP classic pages and for reasons unknown ASP classic locks up on these pages frequently. It's not an individual page, but ALL ASP classic pages lock up. Ah yes, gotta old tech gone bad. It's not super critical since the content is really old, but still a hassle since it's linked content that still gets quite a bit of traffic. When it happens all ASP classic in that AppPool dies. I've been having a hard time tracking this one down - I suspect an errant COM object

I have a Web Monitor running on the server that's checking for failures and while the monitor can detect the failures when the timeouts occur, I didn't have a good way to just restart that particular application pool. I started putzing around with PowerShell, but - as so often seems the case - I can never get the PowerShell syntax right - I just don't use it enough and have to dig out cheat sheets etc. In any case, after about 20 minutes of that I decided to just create a small .NET Console Application that does the trick instead, and in a few minutes I had this:

using System;
using System.Collections.Generic;
using System.Text;
using System.DirectoryServices;

namespace RecycleApplicationPool
{
    class Program
    {

        static void Main(string[] args)
        {
            string appPoolName = "DefaultAppPool";
            string machineName = "LOCALHOST";
            if (args.Length > 0)
                appPoolName = args[0];
            if (args.Length > 1)
                machineName = args[1];

            string error = null;

            DirectoryEntry root = null;
            try
            {
                Console.WriteLine("Restarting Application Pool " + appPoolName + " on " + machineName + "...");
                root = new DirectoryEntry("IIS://" + machineName + "/W3SVC/AppPools/" +appPoolName);
                Console.WriteLine(root.InvokeGet("Name"));
                root.Invoke("Recycle");                
                Console.WriteLine("Application Pool recycling complete...");

            }
            catch(Exception ex)
            {
                error = "Error: Unable to access AppPool: " + ex.Message;                
            }

            
            if ( !string.IsNullOrEmpty(error) )
            {
                Console.WriteLine(error);
                return;
            }
        }
    }
}

To run in you basically provide the name of the ApplicationPool and optionally a machine name if it's not on the local box.

RecyleApplicationPool.exe "WestWindArticles"

And off it goes. What's nice about AppPool recycling versus doing a full IISRESET is that it only affects the AppPool, and more importantly AppPool recycles happen in a staggered fashion - the existing instance isn't shut down immediately until requests finish while a new instance is fired up to handle new requests.

So, now I can easily plug this Executable into my West Wind Web Monitor as an action to take when the site is not responding or timing out which is a big improvement than hanging for an unspecified amount of time.

I'm posting this fairly trivial bit of code just in case somebody (maybe myself a few months down the road) is searching for ApplicationPool recyling code. It's clearly trivial, but I've written batch files for this a bunch of times before and actually having a small utility around without having to worry whether Powershell is installed and configured right is actually an improvement. Next time I think about using PowerShell remind me that it's just easier to just build a small .NET Console app, 'k? :-)

Resources

© Rick Strahl, West Wind Technologies, 2005-2012
Posted in IIS7  .NET  Windows  

Dynamic Code for type casting Generic Types 'generically' in C#

$
0
0

C# is a strongly typed language and while that's a fundamental feature of the language there are more and more situations where dynamic types make a lot of sense. I've written quite a bit about how I use dynamic for creating new type extensions:

Today I want to point out an example of a much simpler usage for dynamic that I use occasionally to get around potential static typing issues in C# code especially those concerning generic types.

TypeCasting Generics

Generic types have been around since .NET 2.0 and they can be extremely useful in creating flexible class designs that are extensible and can deal with different member types/elements. Most of the time they provide great enhancements, but dealing with casting in generics can become very complex especially if there are interdependencies in object.

I've run into a number of situations in the past - especially with generic types that don't implement specific interfaces that can be cast to - where I've been unable to properly cast an object when it's passed to a method or assigned to a property. Granted often this can be a sign of bad design, but in at least some situations the code that needs to be integrated is not under my control so I have to make due with what's available, or the parent object is too complex or intermingled to be easily refactored to a new usage scenario.

Here's an example that I ran into in my own RazorHosting library - so I have really no excuse, but I also don't see another clean way around it in this case.

A Generic Example

Imagine I've implemented a generic type like this:

    public class RazorEngine<TBaseTemplateType> 
where TBaseTemplateType : RazorTemplateBase, new()

You can now happily instantiate new generic versions of this type with custom template bases or even a non-generic version which is implemented like this:

    public class RazorEngine : RazorEngine<RazorTemplateBase>
    {
        public RazorEngine() : base() { }        
    }

To instantiate one:

   var engine = new RazorEngine<MyCustomRazorTemplate>();

Now imagine that the template class receives a reference to the engine when it's instantiated. This code is fired as part of the Engine pipeline when it gets ready to execute the template. It instantiates the template and assigns itself to the template:

    var template = new TBaseTemplateType() {
       Engine = this

}

The problem here is that possibly many variations of RazorEngine<T> can be passed to Engine. I can have RazorTemplateBase, RazorFolderHostTemplateBase, CustomRazorTemplateBase etc. as generic parameters and the Engine property has to reflect that somehow. Additionally the generic parameter TBaseTemplateType is in fact the actual active template class. I know what the type is, but not how to declare it on the Engine property to reflect the semi dynamic nature based on inheritance.

So, how would I cast that? My first inclination was to use an interface on the engine class and then cast to the interface.  Generally that works, but unfortunately here the engine class is generic and has a few members that require the template type in the member signatures. So while I certainly can implement an interface:

   public interface IRazorEngine<TBaseTemplateType>

it doesn't really help for passing this generically templated object to the template class - I still can't cast it if multiple differently typed versions of the generic type could be passed. I have the exact same issue in that I can't specify a 'generic' generic parameter, since there's no underlying base type that's common. Even if I could manage to create a non-generic interface, there are other issues such as instantiation and passing across AppDomain boundaries - this also wouldn't work.

In light of this I decided on using object and the following syntax for the property (and the same would be true for a method parameter):

  public class RazorTemplateBase :MarshalByRefObject,IDisposable
  {
       public object Engine {get;set; }
  }

Now because the Engine property is a non-typed object, when I need to do something with this value, I still have no way to cast it explicitly. Specifically the cast needs to be RazorEngine<CurrentRunningClass> which of course doesn't work because that would require a runtime setting.

What I would be nice is:

  public RazorEngine<> Engine { get; set; }

but that's not possible.

 

Dynamic to the Rescue

Luckily with the dynamic type this sort of thing can be mitigated fairly easily.

For example here's a method that uses the Engine property and uses the well known class interface by simply casting the plain object reference to dynamic and then firing away on the properties and methods of the base template class that are common to all templates:

/// <summary>
/// Allows rendering a dynamic template from a string template
/// passing in a model.   This is like rendering a partial
/// but providing the input as a 
/// </summary>
public virtual string RenderTemplate(string template,object model)
{            
    if (template == null)
        return string.Empty;

    // if there's no template markup
    if(!template.Contains("@"))
        return template;

    // use dynamic to get around generic type casting
    dynamic engine = Engine;
    string result = engine.RenderTemplate(template, model);
    if (result == null)
        throw new ApplicationException("RenderTemplate failed: " + engine.ErrorMessage);
                       
    return result;
}

Prior to .NET 4.0  I would have had to use Reflection for this sort of thing which would have a been a heck of a lot more verbose, but dynamic makes this so much easier and cleaner and in this case at least the overhead is negliable since it's a single dynamic operation on an otherwise very complex operation call.

The point is: To fix this properly with static typing would be complex and require some mental gymnastics around generics. Dynamic provides a very easy workaround to make this work without having to redesign several classes and dependencies.

Dynamic as  a Bailout

Sometimes this sort of thing reeks of a design flaw, and I agree that in hindsight this could have been designed differently. But as is often the case this particular scenario wasn't planned for originally and removing the generic signatures from the base type would break a ton of other code in the framework. Given the existing fairly complex engine design, refactoring an interface to remove generic types just to make this particular code work would have been overkill.

Instead dynamic provides a nice and simple and relatively clean solution. Now if there were many other places where this occurs I would probably consider reworking the code to make this cleaner but given this isolated instance and relatively low profile operation use of dynamic seems a valid choice for me.

This solution really works anywhere where you might end up with an inheritance structure that doesn't have a common base or interface that is sufficient. In the example above I know what I'm getting but there's no common base type that I can cast to.

All that said, it's a good idea to think about use of dynamic before you rush in. In many situations there are alternatives that can still work with static typing. Dynamic definitely has some overhead compared to direct static access of objects, so if possible we should definitely stick to static typing.

In the example above the application already uses dynamics extensively for dynamic page page templating and passing models around so introducing dynamics here has very little additional overhead. The operation itself also fires of a fairly resource heavy operation where the overhead of a couple of dynamic member accesses are not a performance issue.

So, what's your experience with dynamic as a bailout mechanism?

© Rick Strahl, West Wind Technologies, 2005-2012
Posted in CSharp  

Caveats with the runAllManagedModulesForAllRequests in IIS 7/8

$
0
0

One of the nice enhancements in IIS 7 (and now 8) is the ability to be able to intercept non-managed - ie. non ASP.NET served - requests from within ASP.NET managed modules. This opened up a ton of new functionality that could be applied across non-managed content using .NET code.

I thought I had a pretty good handle on how IIS 7's Integrated mode pipeline works, but when I put together some samples last tonight I realized that the way that managed and unmanaged requests fire into the pipeline is downright confusing especially when it comes to the runAllManagedModulesForAllRequests attribute. There are a number of settings that can affect whether a managed module receives non-ASP.NET content requests such as static files or requests from other frameworks like PHP or ASP classic, and this is topic of this blog post.

Native and Managed Modules

The integrated mode IIS pipeline for IIS 7 and later - as the name suggests - allows for integration of ASP.NET pipeline events in the IIS request pipeline. Natively IIS runs unmanaged code and there are a host of native mode modules that handle the core behavior of IIS. If you set up a new IIS site or application without managed code support only the native modules are supported and fired without any interaction between native and managed code.

If you use the Integrated pipeline with managed code enabled however things get a little more confusing as there both native modules and .NET managed modules can fire against the same IIS request.

If you open up the IIS Modules dialog you see both managed and unmanaged modules. Unmanaged modules point at physical files on disk, while unmanaged modules point at .NET types and files referenced from the GAC or the current project's BIN folder.

IISModulesInConsole

Both native and managed modules can co-exist and execute side by side and on the same request. When running in IIS 7 the IIS pipeline actually instantiates a the ASP.NET  runtime (via the System.Web.PipelineRuntime class) which unlike the core HttpRuntime classes in ASP.NET receives notification callbacks when IIS integrated mode events fire. The IIS pipeline is smart enough to detect whether managed handlers are attached and if they're none these notifications don't fire, improving performance.

The good news about all of this for .NET devs is that ASP.NET style modules can be used for just about every kind of IIS request. All you need to do is create a new Web Application and enable ASP.NET on it, and then attach managed handlers. Handlers can look at ASP.NET content (ie. ASPX pages, MVC, WebAPI etc. requests) as well as non-ASP.NET content including static content like HTML files, images, javascript and css resources etc. It's very cool that this capability has been surfaced.

However, with that functionality comes a lot of responsibility. Because every request passes through the ASP.NET pipeline if managed modules (or handlers) are attached there are possible performance implications that come with it. Running through the ASP.NET pipeline does add some overhead.

ASP.NET and Your Own Modules

When you create a new ASP.NET project typically the Visual Studio templates create the modules section like this:

  <system.webServer>
    <validation validateIntegratedModeConfiguration="false" />
    <modules runAllManagedModulesForAllRequests="true" >
    </modules>
  </system.webServer> 

Specifically the interesting thing about this is the runAllManagedModulesForAllRequest="true" flag, which seems to indicate that it controls whether any registered modules always run, even when the value is set to false. Realistically though this flag does not control whether managed code is fired for all requests or not. Rather it is an override for the preCondition flag on a particular handler.

With the flag set to the default true setting, you can assume that pretty much every IIS request you receive ends up firing through your ASP.NET module pipeline and every module you have configured is accessed even by non-managed requests like static files. In other words, your module will have to handle all requests. Now so far so obvious.

What's not quite so obvious is what happens when you set the runAllManagedModulesForAllRequest="false". You probably would expect that immediately the non-ASP.NET requests no longer get funnelled through the ASP.NET Module pipeline. But that's not what actually happens.

For example, if I create a module like this:

<add name="SharewareModule" type="HowAspNetWorks.SharewareMessageModule"  />

by default it will fire against ALL requests regardless of the runAllManagedModulesForAllRequests flag. Even if the value runAllManagedModulesForAllRequests="false", the module is fired. Not quite expected.

So what is the runAllManagedModulesForAllRequests really good for? It's essentially an override for managedHandler preCondition. If I declare my handler in web.config like this:

<add name="SharewareModule" type="HowAspNetWorks.SharewareMessageModule" preCondition="managedHandler" />

and the runAllManagedModulesForAllRequests="false" my module only fires against managed requests. If I switch the flag to true, now my module ends up handling all IIS requests that are passed through from IIS.

The moral of the story here is that if you intend to only look at ASP.NET content, you should always set the preCondition="managedHandler" attribute to ensure that only managed requests are fired on this module. But even if you do this, realize that runAllManagedModulesForAllRequests="true" can override this setting.

runAllManagedModulesForAllRequests and Http Application Events

Another place the runAllManagedModulesForAllRequest attribute affects is the Global Http Application object (typically in global.asax) and the Application_XXXX events that you can hook up there. So while the events there are dynamically hooked up to the application class, they basically behave as if they were set with the preCodition="managedHandler" configuration switch.

The end result is that if you have runAllManagedModulesForAllRequests="true" you'll see every Http request passed through the Application_XXXX events, and you only see ASP.NET requests with the flag set to "false".

What's all that mean?

Configuring an application to handle requests for both ASP.NET and other content requests can be tricky especially if you need to mix modules that might require both.

Couple of things are important to remember. If your module doesn't need to look at every request, by all means set a preCondition="managedHandler" on it. This will at least allow it to respond to the runAllManagedModulesForAllRequests="false" flag and then only process ASP.NET requests.

Look really carefully to see whether you actually need runAllManagedModulesForAllRequests="true" in your applications as set by the default new project templates in Visual Studio. Part of the reason, this is the default because it was required for the initial versions of IIS 7 and ASP.NET 2 in order to handle MVC extensionless URLs.

However, if you are running IIS 7 or later and .NET 4.0 you can use the ExtensionlessUrlHandler instead to allow you MVC functionality without requiring runAllManagedModulesForAllRequests="true":

    <handlers>
      <remove name="ExtensionlessUrlHandler-Integrated-4.0" />      
      <add name="ExtensionlessUrlHandler-Integrated-4.0" path="*." 
verb="GET,HEAD,POST,DEBUG,PUT,DELETE,PATCH,OPTIONS"
type="System.Web.Handlers.TransferRequestHandler"
preCondition="integratedMode,runtimeVersionv4.0" /> </handlers>

Oddly this is the default for Visual Studio 2012 MVC template apps, so I'm not sure why the default template still adds runAllManagedModulesForAllRequests="true" is - it should be enabled only if there's a specific need to access non ASP.NET requests.

As a side note, it's interesting that when you access a static HTML resource, you can actually write into the Response object and get the output to show, which is trippy. I haven't looked closely to see how this works - whether ASP.NET just fires directly into the native output stream or whether the static requests are re-routed directly through the ASP.NET pipeline once a managed code module is detected. This doesn't work for all non ASP.NET resources - for example, I can't do the same with ASP classic requests, but it makes for an interesting demo when injecting HTML content into a static HTML page :-)

Note that on the original Windows Server 2008 and Vista (IIS 7.0) you might need a HotFix in order for ExtensionLessUrlHandler to work properly for MVC projects. On my live server I needed it (about 6 months ago), but others have observed that the latest service updates have integrated this functionality and the hotfix is not required. On IIS 7.5 and later I've not needed any patches for things to just work.

Plan for non-ASP.NET Requests

It's important to remember that if you write a .NET Module to run on IIS 7, there's no way for you to prevent non-ASP.NET requests from hitting your module. So make sure you plan to support requests to extensionless URLs, to static resources like files. Luckily ASP.NET creates a full Request and full Response object for you for non ASP.NET content. So even for static files and even for ASP classic for example, you can look at Request.FilePath or Request.ContentType (in post handler pipeline events) to determine what content you are dealing with.

As always with Module design make sure you check for the conditions in your code that make the module applicable and if a filter fails immediately exit - minimize the code that runs if your module doesn't need to process the request.

© Rick Strahl, West Wind Technologies, 2005-2012
Posted in IIS7   ASP.NET  

Back to Basics: When does a .NET Assembly Dependency get loaded

$
0
0

When we work on typical day to day applications, it's easy to forget some of the core features of the .NET framework. For me personally it's been a long time since I've learned about some of the underlying CLR system level services even though I rely on them on a daily basis. I often think only about high level application constructs and/or high level framework functionality, but the low level stuff is often just taken for granted.

Over the last week at DevConnections I had all sorts of low level discussions with other developers about the inner workings of this or that technology (especially in light of my Low Level ASP.NET Architecture talk and the Razor Hosting talk). One topic that came up a couple of times and ended up a point of confusion even amongst some seasoned developers (including some folks from Microsoft <snicker>) is when assemblies actually load into a .NET process.

There are a number of different ways that assemblies are loaded in .NET. When you create a typical project, assemblies usually come from:

  • The Assembly reference list of the top level 'executable' project
  • The Assembly references of referenced projects
  • Dynamically loaded assemblies, using runtime loading via AppDomain or Reflection loading

In addition .NET automatically loads mscorlib (most of the System namespace) as part of the .NET runtime hosting process that hoists up the .NET runtime in EXE apps, or some other kind of runtime hosting environment (runtime hosting in servers like IIS, SQL Server or COM Interop). In hosting environments the runtime host may also pre-load a bunch of assemblies on its own (for example the ASP.NET host requires all sorts of assemblies just to run itself, before ever routing into your user specific code).

Assembly Loading

The most obvious source of loaded assemblies is the top level application's assembly reference list. You can add assembly references to a top level application and those assembly references are then available to the application.

In a nutshell, referenced assemblies are not immediately loaded - they are loaded on the fly as needed. So regardless of whether you have an assembly reference in a top level project, or a dependent assembly assemblies typically load on an as needed basis, unless explicitly loaded by user code. The same is true of dependent assemblies.

To check this out I ran a simple test: I have a utility assembly Westwind.Utilities which is a general purpose library that can work in any type of project. Due to a couple of small requirements for encoding and a logging piece that allows logging Web content (dependency on HttpContext.Current) this utility library has a dependency on System.Web. Now System.Web is a pretty large assembly and generally you'd want to avoid adding it to a non-Web project if it can be helped.

So I created a Console Application that loads my utility library:

AssemblyListInProjects

You can see that the top level Console app a reference to Westwind.Utilities and System.Data (beyond the core .NET libs). The Westwind.Utilities project on the other hand has quite a few dependencies including System.Web.

I then add a main program that accesses only a simple utillity method in the Westwind.Utilities library that doesn't require any of the classes that access System.Web:

static void Main(string[] args)
        {Console.WriteLine(StringUtils.NewStringId());            Console.ReadLine();
        }

StringUtils.NewStringId() calls into Westwind.Utilities, but it doesn't rely on System.Web.

Any guesses what the assembly list looks like when I stop the code on the ReadLine() command?

I'll wait here while you think about it…

So, when I stop on ReadLine() and then fire up Process Explorer and check the assembly list I get:

AssembliesProcessExplorer

We can see here that .NET has not actually loaded any of the dependencies of the Westwind.Utilities assembly. Also not loaded is the top level System.Data reference even though it's in the dependent assembly list of the top level project. Since this particular function I called only uses core System functionality (contained in mscorlib) there's in fact nothing else loaded beyond the main application and my Westwind.Utilities assembly that contains the method accessed. None of the dependencies of Westwind.Utilities loaded.

If you were to open the assembly in a disassembler like Reflector or ILSpy, you would however see all the compiled in dependencies. The referenced assemblies are in the dependency list and they are loadable, but they are not immediately loaded by the application.

In other words the C# compiler and .NET linker are smart enough to figure out the dependencies based on the code that actually is referenced from your application and any dependencies cascading down into the dependencies from your top level application into the referenced assemblies. In the example above the usage requirement is pretty obvious since I'm only calling a single static method and then exiting the app, but in more complex applications these dependency relationships become very complicated - however it's all taken care of by the compiler and linker figuring out what types and members are actually referenced and including only those assemblies that are in fact referenced in your code or required by any of your dependencies.

The good news here is: That if you are referencing an assembly that has a dependency on something like System.Web in a few places that are not actually accessed by any of your code or any dependent assembly code that you are calling, that assembly is never loaded into memory!

Some Hosting Environments pre-load Assemblies

The load behavior can vary however. In Console and desktop applications we have full control over assembly loading so we see the core CLR behavior. However other environments like ASP.NET for example will preload referenced assemblies explicitly as part of the startup process - primarily to minimize load conflicts. Specifically ASP.NET pre-loads all assemblies referenced in the GAC assembly list and the /bin folder. So in Web applications it definitely pays to minimize your top level assemblies if they are not used.

Understanding when Assemblies Load

To clarify and see it actually happen what I described in the first example , let's look at a couple of other scenarios. To see assemblies loading at runtime in real time lets create a utility function to print out loaded assemblies to the console:

public static void PrintAssemblies()
    {var assemblies = AppDomain.CurrentDomain.GetAssemblies();foreach (var assembly in assemblies)
        {Console.WriteLine(assembly.GetName());
        }
    }

Now let's look at the first scenario where I have class method that references internally uses System.Web. In the first scenario lets add a method to my main program like this:

static void Main(string[] args)
        {Console.WriteLine(StringUtils.NewStringId());Console.ReadLine();
PrintAssemblies(); }
public static void WebLogEntry() {var entry = new WebLogEntry(); entry.UpdateFromRequest();Console.WriteLine(entry.QueryString); }

UpdateFromWebRequest() internally accesses HttpContext.Current to read some information of the ASP.NET Request object so it clearly needs a reference System.Web to work. In this first example, the method that holds the calling code is never called, but exists as a static method that can potentially be called externally at some point.

What do you think will happen here with the assembly loading? Will System.Web load in this example?

Console1

No - it doesn't. Because the WebLogEntry() method is never called by the mainline application (or anywhere else) System.Web is not loaded. .NET dynamically loads assemblies as code that needs it is called. No code references the WebLogEntry() method and so System.Web is never loaded.

Next, let's add the call to this method, which should trigger System.Web to be loaded because a dependency exists. Let's change the code to:

static void Main(string[] args)
    {Console.WriteLine(StringUtils.NewStringId());Console.WriteLine("--- Before:");
        PrintAssemblies();WebLogEntry();Console.WriteLine("--- After:");
        PrintAssemblies();Console.ReadLine();
    }public static void WebLogEntry()
    {var entry = new WebLogEntry();entry.UpdateFromRequest();Console.WriteLine(entry.QueryString);
    }

Looking at the code now, when do you think System.Web will be loaded? Will the before list include it?

Console2

Yup System.Web gets loaded, but only after it's actually referenced. In fact, just until before the call to UpdateFromRequest() System.Web is not loaded - it only loads when the method is actually called and requires the reference in the executing code.

Assembly Unloading

As a side not, when an assembly is loaded in .NET, it loads into an AppDomain, and can never be unloaded from that AppDomain. That's as in never ever. Meaning once you take the memory hit from the assembly loading that memory can never release. The only way .NET assemblies can be unloaded is by unloading the AppDomain. If you have applications that need to dynamically load assemblies (like a hosting or scripting engine or plug-ins for example), it's a good idea to load assemblies into a separate AppDomain that can be unloaded when you're done, or optionally allows occasional unloading to minimize memory usage.

The fact that you can't unload assemblies is one of the reasons why discussions about assembly loading and trying to avoid loading unnecessary stuff usually comes up in the first place :-)

Moral of the Story

So what have we learned - or maybe remembered again?

  • Dependent Assembly References are not pre-loaded when an application starts (by default)
  • Dependent Assemblies that are not referenced by executing code are never loaded
  • Dependent Assemblies are just in time loaded when first referenced in code
  • Once Assemblies are loaded they can never be unloaded, unless the AppDomain that host them is unloaded.

All of this is nothing new - .NET has always worked like this. But it's good to have a refresher now and then and go through the exercise of seeing it work in action. It's not one of those things we think about everyday, and as I found out last week, I couldn't remember exactly how it worked since it's been so long since I've learned about this. And apparently I'm not the only one as several other people I had discussions with in relation to loaded assemblies also didn't recall exactly what should happen or assumed incorrectly that just having a reference automatically loads the assembly.

The moral of the story for me is: Trying at all costs to eliminate an assembly reference from a component is not quite as important as it's often made out to be.

For example, the Westwind.Utilities module described above has a logging component, including a Web specific logging entry that supports pulling information from the active HTTP Context. Adding that feature requires a reference to System.Web. Should I worry about this in the scope of this library? Probably not, because if I don't use that one class of nearly a hundred, System.Web never gets pulled into the parent process. IOW, System.Web only loads when I use that specific feature and if I am, well I clearly have to be running in a Web environment anyway to use it realistically. The alternative would be considerably uglier: Pulling out the WebLogEntry class and sticking it into another assembly and breaking up the logging code. In this case - definitely not worth it.

So, .NET definitely goes through some pretty nifty optimizations to ensure that it loads only what it needs and in most cases you can just rely on .NET to do the right thing. Sometimes though assembly loading can go wrong (especially when signed and versioned local assemblies are involved), but that's subject for a whole other post…

© Rick Strahl, West Wind Technologies, 2005-2012
Posted in .NET  CSharp  

HTML5 Input type=date Formatting Issues

$
0
0

One of the nice features in HTML5 is the abililty to specify a specific input type for HTML text input boxes. There a host of very useful input types available including email, number, date, datetime, month, number, range, search, tel, time, url and week. For a more complete list you can check out the MDN reference. Date input types also support automatic validation which can be useful in some scenarios but maybe can get in the way at other times.

One of the more common input types, and one that can most benefit of a custom UI for selection is of course date input. Almost every application could use a decent date representation and HTML5's date input type seems to push into the right direction. It'd be nice if you could just say:

<form action="DateTest.html"><label for="FromDate">Enter a Date:</label>
<
input type="date" id="FromDate" name="FromDate" value="11/08/2012" class="date" />
<hr /><input type="submit" id="btnSubmit" name="btnSubmit" value="Save Date" class="smallbutton" /></form>

but if you'd expect to just work, you're likely to be pretty disappointed.

Problem #1: Browser Support

For starters there's browser support. Out of the major browsers only the latest versions of WebKit and Opera based browsers seem to support date input. Neither FireFox, nor any version of Internet Explorer (including the new touch enabled IE10 in Windows RT) support input type=date. Browser support is an issue, but it would be OK if it wasn't for problem #2.

Problem #2: Date Formatting

If you look at my date input from before:

<input type="date" id="FromDate" name="FromDate" value="11/08/2012" class="date" />

You can see that my date is formatted in local date format (ie. en-us). Now when I run this sadly the form that comes up in Chrome (and also iOS mobile browsers) comes up like this:

EmptyDateBox

Chrome isn't recognizing my local date string. Instead it's expecting my date format to be provided in ISO 8601 format which is:

2012-11-08

So if I change the date input field to:

<input type="date" id="FromDate" name="FromDate" value="2012-10-08" class="date" />
I correctly get the date field filled in:
DateWithISO

Also when I pick a date with the DatePicker the date value is also returned is also set to the ISO date format. Yet notice how the date is still formatted to the local date time format (ie. en-US format). So if I pick a new date:

DatePickingChrome

and then save, the value field is set back to:

2012-11-15

using the ISO format. The same is true for Opera and iOS browsers and I suspect any other WebKit style browser and their date pickers.

So to summarize input type=date:

  • Expects ISO 8601 format dates to display intial values
  • Sets selected date values to ISO 8601

Now what?

This would sort of make sense, if all browsers supported input type=date. It'd be easy because you could just format dates appropriately when you set the date value into the control by applying the appropriate culture formatting (ie. .ToString("yyyy-MM-dd") ). .NET is actually smart enough to pick up the date on the other end for modelbinding when ISO 8601 is used. For other environments this might be a bit more tricky.

input type=date is clearly the way to go forward. Date controls implemented in HTML are going the way of the dodo, given the intricacies of mobile platforms and scaling for both desktop and mobile. I've been using jQuery UI Datepicker for ages but once going to mobile, that's no longer an option as the control doesn't scale down well for mobile apps (at least not without major re-styling). It also makes a lot of sense for the browser to provide this functionality - creating a consistent date input experience across apps only makes sense, which is why I find it baffling that neither FireFox nor IE 10 deign it necessary to support date input natively.

The problem is that a large number of even the latest and greatest browsers don't support this. So now you're stuck with not knowing what date format you have to serve since neither the local format, nor the ISO format works in all cases.

Modernizr

It's possible to check for the availability of input types with Modernizr by checking:

Modernizr.inputtypes.date

for true or false. Unfortunately this is a client side thing, which means you'd have to delay formatting your date on the client using some sort of client library for date formatting like Moment.js rather than pushing a date into the UI from the server. This would work for a client heavy application that's using client side templates to render content, but not so well for a typical HTML server rendered app.

Compromise?

For my current app I just broke down and used the ISO format and so I'll live with the non-local date format.

<input type="date" id="ToDate" name="ToDate" value="2012-11-08" class="date"/>

Here's what this looks like on Chrome:

ChromeInput

Here's what it looks like on my iPhone:

iphonedate

Both Chrome and the phone do this the way it should be. The iPhone date display and date picker in particular are very nice (better than anything on the desktop - that's for sure) and it demonstrates why we'd want this implemented in the browser to share this common UI for any application that provides date input. The iOS built-in date picker there certainly beats manually trying to edit the date using finger gymnastics.

Finally here's what the date looks like in FireFox which doesn't support date input types:

FireFoxISODateString

Certainly this is not the ideal date format, but it's clear enough I suppose. If users enter a date in local US format and that works as well (but won't work for other locales). It'll have to do.

Over time one can only hope that other browsers will finally decide to implement this functionality natively to provide a unique experience. Until then, incomplete solutions it is.

Related Posts

© Rick Strahl, West Wind Technologies, 2005-2012
Posted in HTML5  HTML  

DevConnections Session Slides, Samples and Links

$
0
0

Finally coming up for air this week, after catching up with being on the road for the better part of three weeks. Here are my slides, samples and links for my four DevConnections Session two weeks ago in Vegas. I ended up doing one extra un-prepared for session on WebAPI and AJAX, as some of the speakers were either delayed or unable to make it at all to Vegas due to Sandy's mayhem. It was pretty hectic in the speaker room as Erik (our event coordinator extrodinaire) was scrambling to fill session slots with speakers :-). Surprisingly it didn't feel like the storm affected attendance drastically though, but I guess it's hard to tell without actual numbers.

The conference was a lot of fun - it's been a while since I've been speaking at one of these larger conferences. I'd been taking a hiatus, and I forgot how much I enjoy actually giving talks. Preparing - well not  quite so much, especially since I ended up essentially preparing or completely rewriting for all three of these talks and I was stressing out a bit as I was sick the week before the conference and didn't get as much time to prepare as I wanted to. But - as always seems to be the case - it all worked out, but I guess those that attended have to be the judge of that…

It was great to catch up with my speaker friends as well - man I feel out of touch. I got to spend a bunch of time with Dan Wahlin, Ward Bell, Julie Lerman and for about 10 minutes even got to catch up with the ever so busy Michele Bustamante. Lots of great technical discussions including a fun and heated REST controversy with Ward and Howard Dierking. There were also a number of great discussions with attendees, describing how they're using the technologies touched in my talks in live applications. I got some great ideas from some of these and I wish there would have been more opportunities for these kinds of discussions. One thing I miss at these Vegas events though is some sort of coherent event where attendees and speakers get to mingle. These Vegas conferences are just like "go to sessions, then go out and PARTY on the town" - it's Vegas after all! But I think that it's always nice to have at least one evening event where everybody gets to hang out together and trade stories and geek talk. Overall there didn't seem to be much opportunity for that beyond lunch or the small and short exhibit hall events which it seemed not many people actually went to.

Anyways, a good time was had. I hope those of you that came to my sessions learned something useful. There were lots of great questions and discussions after the sessions - always appreciate hearing the real life scenarios that people deal with in relation to the abstracted scenarios in sessions.

Here are the Session abstracts, a few comments and the links for downloading slides and  samples. It's not quite like being there, but I hope this stuff turns out to be useful to some of you. I'll be following up a couple of these sessions with white papers in the following weeks.

Enjoy.

ASP.NET Architecture: How ASP.NET Works at the Low Level

Abstract:
Interested in how ASP.NET works at a low level? ASP.NET is extremely powerful and flexible technology, but it's easy to forget about the core framework that underlies the higher level technologies like ASP.NET MVC, WebForms, WebPages, Web Services that we deal with on a day to day basis. The ASP.NET core drives all the higher level handlers and frameworks layered on top of it and with the core power comes some complexity in the form of a very rich object model that controls the flow of a request through the ASP.NET pipeline from Windows HTTP services down to the application level. To take full advantage of it, it helps to understand the underlying architecture and model. This session discusses the architecture of ASP.NET along with a number of useful tidbits that you can use for building and debugging your ASP.NET applications more efficiently. We look at overall architecture, how requests flow from the IIS (7 and later) Web Server to the ASP.NET runtime into HTTP handlers, modules and filters and finally into high-level handlers like MVC, Web Forms or Web API. Focus of this session is on the low-level aspects on the ASP.NET runtime, with examples that demonstrate the bootstrapping of ASP.NET, threading models, how Application Domains are used, startup bootstrapping, how configuration files are applied and how all of this relates to the applications you write either using low-level tools like HTTP handlers and modules or high-level pages or services sitting at the top of the ASP.NET runtime processing chain.

Comments:
I was surprised to see so many people show up for this session - especially since it was the last session on the last day and a short 1 hour session to boot. The room was packed and it was to see so many people interested the abstracts of architecture of ASP.NET beyond the immediate high level application needs. Lots of great questions in this talk as well - I only wish this session would have been the full hour 15 minutes as we just a little short of getting through the main material (didn't make it to Filters and Error handling).

I haven't done this session in a long time and I had to pretty much re-figure all the system internals having to do with the ASP.NET bootstrapping in light for the changes that came with IIS 7 and later. The last time I did this talk was with IIS6, I guess it's been a while.

I love doing this session, mainly because in my mind the core of ASP.NET overall is so cleanly designed to provide maximum flexibility without compromising performance that has clearly stood the test of time in the 10 years or so that .NET has been around. While there are a lot of moving parts, the technology is easy to manage once you understand the core components and the core model hasn't changed much even while the underlying architecture that drives has been almost completely revamped especially with the introduction of IIS 7 and later.

 

Introduction to using jQuery with ASP.NET

Abstract:
In this session you'll learn how to take advantage of jQuery in your ASP.NET applications. Starting with an overview of jQuery client features via many short and fun examples, you'll find out about core features like the power of selectors for document element selection, manipulating these elements with jQuery's wrapped set methods in a browser independent way, how to hook up and handle events easily and generally apply concepts of unobtrusive JavaScript principles to client scripting. The second half of the session then delves into jQuery's AJAX features and several different ways how you can interact with ASP.NET on the server. You'll see examples of using ASP.NET MVC for serving HTML and JSON AJAX content, as well as using the new ASP.NET Web API to serve JSON and hypermedia content. You'll also see examples of client side templating/databinding with Handlebars and Knockout.

Comments:
This session was in a monster of a room and to my surprise it was nearly packed, given that this was a 100 level session. I can see that it's a good idea to continue to do intro sessions to jQuery as there appeared to be quite a number of folks who had not worked much with jQuery yet and who most likely could greatly benefit from using it. Seemed seemed to me the session got more than a few people excited to going if they hadn't yet :-).  Anyway I just love doing this session because it's mostly live coding and highly interactive - not many sessions that I can build things up from scratch and iterate on in an hour. jQuery makes that easy though.

Resources:

 

Hosting the Razor Scripting Engine in Your Own Applications

Abstract:
The Razor Engine used in ASP.NET MVC and ASP.NET Web Pages is a free-standing scripting engine that can be disassociated from these Web-specific implementations and can be used in your own applications. Razor allows for a powerful mix of code and text rendering that makes it a wonderful tool for any sort of text generation, from creating HTML output in non-Web applications, to rendering mail merge-like functionality, to code generation for developer tools and even as a plug-in scripting engine. In this session, we'll look at the components that make up the Razor engine and how you can bootstrap it in your own applications to hook up templating. You'll find out how to create custom templates and manage Razor requests that can be pre-compiled, detecting page changes and act in ways similar to a full runtime. We look at ways that you can pass data into the engine and retrieve both the rendered output as well as result values in a package that makes it easy to plug Razor into your own applications.

Comments:
That this session was picked was a bit of a surprise to me, since it's a bit of a niche topic. Even more of a surprise was that during the session quite a few people who attended had actually used Razor externally and were there to find out more about how the process works and how to extend it. In the session I talk a bit about a custom Razor hosting implementation (Westwind.RazorHosting) and drilled into the various components required to build a custom Razor Hosting engine and a runtime around it. This sessions was a bit of a chore to prepare for as there are lots of technical implementation details that needed to be dealt with and squeezing that into an hour 15 is a bit tight (and that aren't addressed even by some of the wrapper libraries that exist). Found out though that there's quite a bit of interest in using a templating engine outside of web applications, or often side by side with the HTML output generated by frameworks like MVC or WebForms.

An extra fun part of this session was that this was my first session and when I went to set up I realized I forgot my mini-DVI to VGA adapter cable to plug into the projector in my room - 6 minutes before the session was about to start. So I ended up sprinting the half a mile + back to my room - and back at a full sprint. I managed to be back only a couple of minutes late, but when I started I was out of breath for the first 10 minutes or so, while trying to talk. Musta sounded a bit funny as I was trying to not gasp too much :-)

Resources:

 

Introduction to ASP.NET Web API for AJAX Applications

Abstract:
WebAPI provides a new framework for creating REST based APIs, but it can also act as a backend to typical AJAX operations. This session covers the core features of Web API as it relates to typical AJAX application development. We’ll cover content-negotiation, routing and a variety of output generation options as well as managing data updates from the client in the context of a small Single Page Application style Web app. Finally we’ll look at some of the extensibility features in WebAPI to customize and extend Web API in a number and useful useful ways.

Comments:
This session was a fill in for session slots not filled due MIA speakers stranded by Sandy. I had samples from my previous Web API article so decided to go ahead and put together a session from it. Given that I spent only a couple of hours preparing and putting slides together I was glad it turned out as it did - kind of just ran itself by way of the examples I guess as well as nice audience interactions and questions. Lots of interest - and also some confusion about when Web API makes sense.

Both this session and the jQuery session ended up getting a ton of questions about when to use Web API vs. MVC, whether it would make sense to switch to Web API for all AJAX backend work etc. In my opinion there's no need to jump to Web API for existing applications that already have a good AJAX foundation. Web API is awesome for real externally consumed APIs and clearly defined application AJAX APIs. For typical application level AJAX calls, it's still a good idea, but ASP.NET MVC can serve most if not all of that functionality just as well. There's no need to abandon MVC (or even ASP.NET AJAX or third party AJAX backends) just to move to Web API. For new projects Web API probably makes good sense for isolation of AJAX calls, but it really depends on how the application is set up. In some cases sharing business logic between the HTML and AJAX interfaces with a single MVC API can be cleaner than creating two completely separate code paths to serve essentially the same business logic.

Resources:

© Rick Strahl, West Wind Technologies, 2005-2012
Posted in Conferences  ASP.NET  

Windows 8 Live Accounts and the actual Windows Account

$
0
0

As if Windows Security wasn't confusing enough, in Windows 8 we get thrown yet another curve ball with Windows Live accounts to logon.

When I set up my Windows 8 machine I originally set it up with a 'real', non-live account that I always use on my Windows machines. I did this mainly so I have a matching account for resources around my home and intranet network so I could log on to network resources properly. At some point later I decided to set up Windows Live security just to see how changes things.

Windows wants you to use Windows Live

Windows 8 logins are required in order for the Windows RT account info to work. Not that I care - since installing Windows 8 I've maybe spent 10 minutes with Windows RT because - well it's pretty freaking sucky on the desktop. From shitty apps to mis-managed screen real estate I can't say that there's anything compelling there to date, but then I haven't looked that hard either.

Anyway… I set up the Windows Live account to see if that changes things. It does - I do get all my live logins to work from Live Account so that Twitter and Facebook posts and pictures and calendars all show up on live tiles on the start screen and in the actual apps. That's nice-ish, but hardly that exciting given that all of the apps tied to those live tiles are average at best. And it would have been nice if all of this could be done without being forced into running with a Windows Live User Account - this all feels like strong-arming you into moving into Microsofts walled garden… and that's probably what it's meant to do.

Who am I?

The real problem to me though is that these Windows Live and raw Windows User accounts are a bit unpredictable especially when it comes to developer information about the account and which credentials to use.

So for example Windows reports folder security like this:

SecurityDialog

Notice it's showing my Windows Live account. Now if I go to Edit and try to add my Windows user account (rstrahl) it'll just automatically show up as the live account.

On the other hand though the underlying system sees everything as my real Windows account. After I switched to a Windows Live login account and I have to login to Windows with my Live account, what do you suppose this returns?

Console.WriteLine(Environment.UserName);

It returns my raw Windows user account (rstrahl). All my permissions, all my actual settings and the desktop console altogether run under that account. If I look in TaskManager (or Process Explorer for me) I see:

ProcExplorer

Everything running on the desktop shell with my login running under my Windows user account.

I suppose it makes sense, but where is that association happening? When I switched to a Windows Live account, nowhere did I associate my real account with the Live account - it just happened. And looking through the account configuration dialogs I can't find any reference to the raw Windows account. Other than switching back I see no mention anywhere of the raw Windows account - everything refers to the Live account.

Right then, clear as potato soup!

So this is who you really are!

The problem is that in some situations this schizophrenic account behavior gets a bit weird. Today I was running a local Web application in IIS that uses Windows Authentication - I tried to log-in with my real Windows account login because that's what I'm used to using with WINDOWS freaking Authentication through IIS. But… it failed. I checked my IIS settings, my apps login settings and I just could not for the life of me get into the site with my Windows username.

That is until I finally realized that I should try using my Windows Live credentials instead. And that worked. So now in this Windows Authentication dialog I had to type in my Live ID and password, which is - just weird. Then in IIS if I look at a Trace page (or in my case my app's Status page) I see that the logged on account is - my Windows user account.

What's really annoying about this is that in some places it uses the live account in other places it uses my Windows account. If I remote desktop into my Web server online - I have to use the local authentication dialog but I have to put in my real Windows credentials not the Live account.

Oh yes, it's all so terribly intuitive and logical…

So in summary, when you log on with a Live account you are actually mapped to an underlying Windows user. In any application if you check the user name it'll be the underlying user account (not sure what happens in a Windows RT app or even what mechanism is used there to get the user name info).  When logging on to local machine resource with user name and password you have to use your Live IDs even if the permissions on the resources are mapped to your underlying Windows account.

Easy enough I suppose, but still not exactly intuitive behavior…

© Rick Strahl, West Wind Technologies, 2005-2012
Posted in Windows  

WCF WS-Security and WSE Nonce Authentication

$
0
0

WCF makes it fairly easy to access WS-* Web Services, except when you run into a service format that it doesn't support. Even then WCF provides a huge amount of flexibility to make the service clients work, however finding the proper interfaces to make that happen is not easy to discover and for the most part undocumented unless you're lucky enough to run into a blog, forum or StackOverflow post on the matter.

This is definitely true for the Password Nonce as part of the WS-Security/WSE protocol, which is not natively supported in WCF. Specifically I had a need to create a WCF message on the client that includes a WS-Security header that looks like this from their spec document:

<soapenv:Header><wsse:Security soapenv:mustUnderstand="1"xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"><wsse:UsernameToken wsu:Id="UsernameToken-8"xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"><wsse:Username>TeStUsErNaMe1</wsse:Username><wsse:Password Type="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText"
>TeStPaSsWoRd1</wsse:Password><wsse:Nonce EncodingType="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-soap-message-security-1.0#Base64Binary"
>f8nUe3YupTU5ISdCy3X9Gg==</wsse:Nonce><wsu:Created>2011-05-04T19:01:40.981Z</wsu:Created></wsse:UsernameToken></wsse:Security></soapenv:Header>

Specifically, the Nonce and Created keys are what WCF doesn't create or have a built in formatting for.

Why is there a nonce? My first thought here was WTF? The username and password are there in clear text, what does the Nonce accomplish? The Nonce and created keys are are part of WSE Security specification and are meant to allow the server to detect and prevent replay attacks. The hashed nonce should be unique per request which the server can store and check for before running another request thus ensuring that a request is not replayed with exactly the same values.

Basic ServiceUtl Import - not much Luck

The first thing I did when I imported this service with a service reference was to simply import it as a Service Reference. The Add Service Reference import automatically detects that WS-Security is required and appropariately adds the WS-Security to the basicHttpBinding in the config file:

<?xml version="1.0" encoding="utf-8" ?><configuration><system.serviceModel><bindings><basicHttpBinding><binding name="RealTimeOnlineSoapBinding"><security mode="Transport" /></binding><binding name="RealTimeOnlineSoapBinding1" /></basicHttpBinding></bindings><client><endpoint address="https://notarealurl.com:443/services/RealTimeOnline"binding="basicHttpBinding" bindingConfiguration="RealTimeOnlineSoapBinding"contract="RealTimeOnline.RealTimeOnline" name="RealTimeOnline" /></client></system.serviceModel></configuration>

If if I run this as is using code like this:

var client = new RealTimeOnlineClient();

client.ClientCredentials.UserName.UserName = "TheUsername";
client.ClientCredentials.UserName.Password = "ThePassword";

I get nothing in terms of WS-Security headers. The request is sent, but the the binding expects transport level security to be applied, rather than message level security. To fix this so that a WS-Security message header is sent the security mode can be changed to:

<security mode="TransportWithMessageCredential" />

Now if I re-run I at least get a WS-Security header which looks like this:

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"xmlns:u="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"><s:Header><o:Security s:mustUnderstand="1"xmlns:o="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"><u:Timestamp u:Id="_0"><u:Created>2012-11-24T02:55:18.011Z</u:Created><u:Expires>2012-11-24T03:00:18.011Z</u:Expires></u:Timestamp><o:UsernameToken u:Id="uuid-18c215d4-1106-40a5-8dd1-c81fdddf19d3-1"><o:Username>TheUserName</o:Username><o:Password Type="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText"
>ThePassword</o:Password></o:UsernameToken></o:Security></s:Header>

Closer! Now the WS-Security header is there along with a timestamp field (which might not be accepted by some WS-Security expecting services), but there's no Nonce or created timestamp as required by my original service.

Using a CustomBinding instead

My next try was to go with a CustomBinding instead of basicHttpBinding as it allows a bit more control over the protocol and transport configurations for the binding. Specifically I can explicitly specify the message protocol(s) used. Using configuration file settings here's what the config file looks like:

<?xml version="1.0"?><configuration><system.serviceModel><bindings><customBinding><binding name="CustomSoapBinding"><security includeTimestamp="false"authenticationMode="UserNameOverTransport"defaultAlgorithmSuite="Basic256"requireDerivedKeys="false"messageSecurityVersion="WSSecurity10WSTrustFebruary2005WSSecureConversationFebruary2005WSSecurityPolicy11BasicSecurityProfile10"></security><textMessageEncoding messageVersion="Soap11"></textMessageEncoding><httpsTransport maxReceivedMessageSize="2000000000"/></binding></customBinding></bindings><client><endpoint address="https://notrealurl.com:443/services/RealTimeOnline"binding="customBinding"bindingConfiguration="CustomSoapBinding"contract="RealTimeOnline.RealTimeOnline"name="RealTimeOnline" /></client></system.serviceModel><startup><supportedRuntime version="v4.0"sku=".NETFramework,Version=v4.0"/></startup></configuration>

This ends up creating a cleaner header that's missing the timestamp field which can cause some services problems. The WS-Security header output generated with the above looks like this:

<s:Header><o:Security s:mustUnderstand="1"xmlns:o="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"><o:UsernameToken u:Id="uuid-291622ca-4c11-460f-9886-ac1c78813b24-1"><o:Username>TheUsername</o:Username><o:Password Type="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText"
>ThePassword</o:Password></o:UsernameToken></o:Security></s:Header>

This is closer as it includes only the username and password.

The key here is the protocol for WS-Security:

messageSecurityVersion="WSSecurity10WSTrustFebruary2005WSSecureConversationFebruary2005WSSecurityPolicy11BasicSecurityProfile10"

which explicitly specifies the protocol version. There are several variants of this specification but none of them seem to support the nonce unfortunately. This protocol does allow for optional omission of the Nonce and created timestamp provided (which effectively makes those keys optional). With some services I tried that requested a Nonce just using this protocol actually worked where the default basicHttpBinding failed to connect, so this is a possible solution for access to some services.

Unfortunately for my target service that was not an option. The nonce has to be there.

Creating Custom ClientCredentials

As it turns out WCF doesn't have support for the Digest Nonce as part of WS-Security, and so as far as I can tell there's no way to do it just with configuration settings. I did a bunch of research on this trying to find workarounds for this, and I did find a couple of entries on StackOverflow as well as on the MSDN forums. However, none of these are particularily clear and I ended up using bits and pieces of several of them to arrive at a working solution in the end.

The latter forum message is the more useful of the two (the last message on the thread in particular) and it has most of the information required to make this work. But it took some experimentation for me to get this right so I'll recount the process here maybe a bit more comprehensively.

In order for this to work a number of classes have to be overridden:

  • ClientCredentials
  • ClientCredentialsSecurityTokenManager
  • WSSecurityTokenizer

The idea is that we need to create a custom ClientCredential class to hold the custom properties so they can be set from the UI or via configuration settings. The TokenManager and Tokenizer are mainly required to allow the custom credentials class to flow through the WCF pipeline and eventually provide custom serialization.

Here are the three classes required and their full implementations:

public class CustomCredentials : ClientCredentials{public CustomCredentials()
    { }protected CustomCredentials(CustomCredentials cc)
        : base(cc)
    { }public override System.IdentityModel.Selectors.SecurityTokenManager CreateSecurityTokenManager()
    {return new CustomSecurityTokenManager(this);
    }protected override ClientCredentials CloneCore()
    {return new CustomCredentials(this);
    }
}
public class CustomSecurityTokenManager : ClientCredentialsSecurityTokenManager{public CustomSecurityTokenManager(CustomCredentials cred)
        : base(cred)
    { }public override System.IdentityModel.Selectors.SecurityTokenSerializer CreateSecurityTokenSerializer(System.IdentityModel.Selectors.SecurityTokenVersion version)
    {return new CustomTokenSerializer(System.ServiceModel.Security.SecurityVersion.WSSecurity11);
    }
}
public class CustomTokenSerializer : WSSecurityTokenSerializer{public CustomTokenSerializer(SecurityVersion sv)
        : base(sv)
    { }protected override void WriteTokenCore(System.Xml.XmlWriter writer,
                                            System.IdentityModel.Tokens.SecurityToken token)
    {UserNameSecurityToken userToken = token as UserNameSecurityToken;string tokennamespace = "o";DateTime created = DateTime.Now;string createdStr = created.ToString("yyyy-MM-ddThh:mm:ss.fffZ");// unique Nonce value - encode with SHA-1 for 'randomness'
        // in theory the nonce could just be the GUID by itselfstring phrase = Guid.NewGuid().ToString();var nonce = GetSHA1String(phrase);// in this case password is plain text
        // for digest mode password needs to be encoded as:
        // PasswordAsDigest = Base64(SHA-1(Nonce + Created + Password))
        // and profile needs to change to
        //string password = GetSHA1String(nonce + createdStr + userToken.Password);string password = userToken.Password;

        writer.WriteRaw(string.Format("<{0}:UsernameToken u:Id=\"" + token.Id +"\" xmlns:u=\"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd\">" +"<{0}:Username>" + userToken.UserName + "</{0}:Username>" +"<{0}:Password Type=\"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText\">" +
        password + "</{0}:Password>" +"<{0}:Nonce EncodingType=\"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-soap-message-security-1.0#Base64Binary\">" +
        nonce + "</{0}:Nonce>" +"<u:Created>" + createdStr + "</u:Created></{0}:UsernameToken>", tokennamespace));
    }protected string GetSHA1String(string phrase)
    {SHA1CryptoServiceProvider sha1Hasher = new SHA1CryptoServiceProvider();byte[] hashedDataBytes = sha1Hasher.ComputeHash(Encoding.UTF8.GetBytes(phrase));return Convert.ToBase64String(hashedDataBytes);
    }

}

Realistically only the CustomTokenSerializer has any significant code in. The code there deals with actually serializing the custom credentials using low level XML semantics by writing output into an XML writer.

I can't take credit for this code - most of the code comes from the MSDN forum post mentioned earlier - I made a few adjustments to simplify the nonce generation and also added some notes to allow for PasswordDigest generation.

Per spec the nonce is nothing more than a unique value that's supposed to be 'random'. I'm thinking that this value can be any string that's unique and a GUID on its own probably would have sufficed. Comments on other posts that GUIDs can be potentially guessed are highly exaggerated to say the least IMHO. To satisfy even that aspect though I added the SHA1 encryption and binary decoding to give a more random value that would be impossible to 'guess'. The original example from the forum post used another level of encoding and decoding to string in between - but that really didn't accomplish anything but extra overhead.

The header output generated from this looks like this:

<s:Header><o:Security s:mustUnderstand="1"xmlns:o="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"><o:UsernameToken u:Id="uuid-f43d8b0d-0ebb-482e-998d-f544401a3c91-1"xmlns:u="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"><o:Username>TheUsername</o:Username><o:Password Type="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText">ThePassword</o:Password><o:Nonce EncodingType="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-soap-message-security-1.0#Base64Binary"
>PjVE24TC6HtdAnsf3U9c5WMsECY=</o:Nonce><u:Created>2012-11-23T07:10:04.670Z</u:Created></o:UsernameToken></o:Security></s:Header>

which is exactly as it should be.

Password Digest?

In my case the password is passed in plain text over an SSL connection, so there's no digest required so I was done with the code above.

Since I don't have a service handy that requires a password digest,  I had no way of testing the code for the digest implementation, but here is how this is likely to work. If you need to pass a digest encoded password things are a little bit trickier. The password type namespace needs to change to:

http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#Digest

and then the password value needs to be encoded. The format for password digest encoding is this:

Base64(SHA-1(Nonce + Created + Password))

and it can be handled in the code above with this code (that's commented in the snippet above):

string password = GetSHA1String(nonce + createdStr + userToken.Password);

The entire WriteTokenCore method for digest code looks like this:

protected override void WriteTokenCore(System.Xml.XmlWriter writer,
                                        System.IdentityModel.Tokens.SecurityToken token)
{UserNameSecurityToken userToken = token as UserNameSecurityToken;string tokennamespace = "o";DateTime created = DateTime.Now;string createdStr = created.ToString("yyyy-MM-ddThh:mm:ss.fffZ");// unique Nonce value - encode with SHA-1 for 'randomness'
    // in theory the nonce could just be the GUID by itselfstring phrase = Guid.NewGuid().ToString();var nonce = GetSHA1String(phrase);string password = GetSHA1String(nonce + createdStr + userToken.Password);writer.WriteRaw(string.Format("<{0}:UsernameToken u:Id=\"" + token.Id +"\" xmlns:u=\"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd\">" +"<{0}:Username>" + userToken.UserName + "</{0}:Username>" +"<{0}:Password Type=\"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#Digest\">" +    password + "</{0}:Password>" +"<{0}:Nonce EncodingType=\"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-soap-message-security-1.0#Base64Binary\">" +
    nonce + "</{0}:Nonce>" +"<u:Created>" + createdStr + "</u:Created></{0}:UsernameToken>", tokennamespace));        }

I had no service to connect to to try out Digest auth - if you end up needing it and get it to work please drop a comment…

How to use the custom Credentials

The easiest way to use the custom credentials is to create the client in code.

Here's a factory method I use to create an instance of my service client: 

public static RealTimeOnlineClient CreateRealTimeOnlineProxy(string url,string username,string password)
{if (string.IsNullOrEmpty(url))
        url = "https://notrealurl.com:443/cows/services/RealTimeOnline";CustomBinding binding = new CustomBinding();  
var security = TransportSecurityBindingElement.CreateUserNameOverTransportBindingElement(); security.IncludeTimestamp = false; security.DefaultAlgorithmSuite = SecurityAlgorithmSuite.Basic256; security.MessageSecurityVersion =
MessageSecurityVersion.WSSecurity10WSTrustFebruary2005WSSecureConversationFebruary2005WSSecurityPolicy11BasicSecurityProfile10;var encoding = new TextMessageEncodingBindingElement();
    encoding.MessageVersion = MessageVersion.Soap11;var transport = new HttpsTransportBindingElement();
    transport.MaxReceivedMessageSize = 20000000; // 20 megsbinding.Elements.Add(security);
    binding.Elements.Add(encoding);
    binding.Elements.Add(transport);RealTimeOnlineClient client = new RealTimeOnlineClient(binding,new EndpointAddress(url));// to use full client credential with Nonce uncomment this code:
    // it looks like this might not be required - the service seems to work without itclient.ChannelFactory.Endpoint.Behaviors.Remove<System.ServiceModel.Description.ClientCredentials>();
    client.ChannelFactory.Endpoint.Behaviors.Add(new CustomCredentials());
    client.ClientCredentials.UserName.UserName = username;
    client.ClientCredentials.UserName.Password = password;return client;
}

This returns a service client that's ready to call other service methods.

The key item in this code is the ChannelFactory endpoint behavior modification that that first removes the original ClientCredentials and then adds the new one. The ClientCredentials property on the client is read only and this is the way it has to be added.

 

Summary

It's a bummer that WCF doesn't suport WSE Security authentication with nonce values out of the box. From reading the comments in posts/articles while I was trying to find a solution, I found that this feature was omitted by design as this protocol is considered unsecure. While I agree that plain text passwords are rarely a good idea even if they go over secured SSL connection as WSE Security does, there are unfortunately quite a few services (mosly Java services I suspect) that use this protocol. I've run into this twice now and trying to find a solution online I can see that this is not an isolated problem - many others seem to have struggled with this. It seems there are about a dozen questions about this on StackOverflow all with varying incomplete answers. Hopefully this post provides a little more coherent content in one place.

Again I marvel at WCF and its breadth of support for protocol features it has in a single tool. And even when it can't handle something there are ways to get it working via extensibility. But at the same time I marvel at how freaking difficult it is to arrive at these solutions. I mean there's no way I could have ever figured this out on my own. It takes somebody working on the WCF team or at least being very, very intricately involved in the innards of WCF to figure out the interconnection of the various objects to do this from scratch. Luckily this is an older problem that has been discussed extensively online and I was able to cobble together a solution from the online content. I'm glad it worked out that way, but it feels dirty and incomplete in that there's a whole learning path that was omitted to get here…

Man am I glad I'm not dealing with SOAP services much anymore. REST service security - even when using some sort of federation is a piece of cake by comparison :-) I'm sure once standards bodies gets involved we'll be right back in security standard hell…

© Rick Strahl, West Wind Technologies, 2005-2012
Posted in WCF  Web Services  

Set-Cookie Headers getting stripped in ASP.NET HttpHandlers

$
0
0

Yikes, I ran into a real bummer of an edge case yesterday in one of my older low level handler implementations (for West Wind Web Connection in this case). Basically this handler is a connector for a backend Web framework that creates self contained HTTP output. An ASP.NET Handler captures the full output, and then shoves the result down the ASP.NET Response object pipeline writing out the content into the Response.OutputStream and seperately sending the HttpHeaders in the Response.Headers collection.The headers turned out to be the problem and specifically Http Cookies, which for some reason ended up getting stripped out in some scenarios.

My handler works like this: Basically the HTTP response from the backend app would return a full set of HTTP headers plus the content. The ASP.NET handler would read the headers one at a time and then dump them out via Response.AppendHeader().

But I found that in some situations Set-Cookie headers sent along were simply stripped inside of the Http Handler. After a bunch of back and forth with some folks from Microsoft (thanks Damien and Levi!) I managed to pin this down to a very narrow edge scenario.

Note:
This post is based on my setup - it appears that there's something on this particular IIS/project configuration that is causing this as I've not been actually able to create a reproducible scenario in a separate project, which is very strange as I see it in my project on the local machine and on a live site the code is running on. I'm still leaving this post up here as it's a weird scenario that might hit somebody else.

It's easiest to demonstrate the problem with a simple example HttpHandler implementation. The following simulates the very much simplified output generation process that fails in my handler. Specifically I have a couple of headers including a Set-Cookie header and some output that gets written into the Response object.

using System.Web;namespace wwThreads
{public class Handler : IHttpHandler{/* NOTE: 
         * 
         * Run as a web.config set handler (see entry below)
         *
         * Best way is to look at the HTTP Headers in Fiddler
         * or Chrome/FireBug/IE tools and look for the 
         * WWHTREADSID cookie in the outgoing Response headers
         * ( If the cookie is not there you see the problem! )
         */public void ProcessRequest(HttpContext context)
        {                        HttpRequest request = context.Request;HttpResponse response = context.Response;// If ClearHeaders is used Set-Cookie header gets removed!
            // if commented header is sent...response.ClearHeaders();

            response.ClearContent(); 
                                
            // Demonstrate that other headers make itresponse.AppendHeader("RequestId", "asdasdasd");// This cookie gets removed when ClearHeaders above is called
            // When ClearHEaders is omitted above the cookie rendersresponse.AppendHeader("Set-Cookie", "WWTHREADSID=ThisIsThEValue; path=/");// *** This always works, even when explicit 
           //     Set-Cookie above fails and ClearHeaders is called
           //response.Cookies.Add(new HttpCookie("WWTHREADSID", "ThisIsTheValue"));response.Write(@"Output was created.<hr/>
                           Check output…");            
        }public bool IsReusable
        {get { return false; }
        }

    }
}

In order to see the problem behavior this code has to be inside of an HttpHandler, and specifically in a handler defined in web.config with:

<add name=".ck_handler"path="handler.ck"verb="*"type="wwThreads.Handler"preCondition="integratedMode" />

Note: Oddly enough this problem manifests only when configured through web.config, not in an ASHX handler, nor if you paste that same code into an ASPX page or MVC controller.

What's the problem exactly?

The code above simulates the more complex code in my live handler that picks up the HTTP response from the backend application and then peels out the headers and sends them one at a time via Response.AppendHeader. One of the headers in my app can be one or more Set-Cookie.

I found that the Set-Cookie headers were not making it into the Response headers output. Here's the Chrome Http Inspector trace:

NoCookies

Notice, no Set-Cookie header in the Response headers!

Now, running the very same request after removing the call to Response.ClearHeaders() command, the cookie header shows up just fine:

CookieThere

As you might expect it took a while to track this down. At first I thought my backend was not sending the headers but after closer checks I found that indeed the headers were set in the backend HTTP response, and they were indeed getting set via Response.AppendHeader() in the handler code. Yet, no cookie in the output.

In the simulated example the problem is this line:

response.AppendHeader("Set-Cookie", "WWTHREADSID=ThisIsThEValue; path=/");

which in my live code is more dynamic ( ie. AppendHeader(token[0],token[1[]) )as it parses through the headers.

Bizzaro Land: Response.ClearHeaders() causes Cookie to get stripped

Now, here is where it really gets bizarre: The problem occurs only if:

  • Response.ClearHeaders() was called before headers are added
  • It only occurs in Http Handlers declared in web.config

Clearly this is an edge of an edge case but of course - knowing my relationship with Mr. Murphy - I ended up running smack into this problem.

So in the code above if you remove the call to ClearHeaders(), the cookie gets set!  Add it back in and the cookie is not there.

If I run the above code in an ASHX handler it works. If I paste the same code (with a Response.End()) into an ASPX page, or MVC controller it all works. Only in the HttpHandler configured through Web.config does it fail!

Cue the Twilight Zone Music.

Workarounds

As is often the case the fix for this once you know the problem is not too difficult. The difficulty lies in tracking inconsistencies like this down. Luckily there are a few simple workarounds for the Cookie issue.

Don't use AppendHeader for Cookies

The easiest and obvious solution to this problem is simply not use Response.AppendHeader() to set Cookies. Duh! Under normal circumstances in application level code there's rarely a reason to write out a cookie like this:

response.AppendHeader("Set-Cookie", "WWTHREADSID=ThisIsThEValue; path=/");

but rather create the cookie using the Response.Cookies collection:

response.Cookies.Add(new HttpCookie("WWTHREADSID", "ThisIsTheValue"));

Unfortunately, in my case where I dynamically read headers from the original output and then dynamically  write header key value pairs back  programmatically into the Response.Headers collection, I actually don't look at each header specifically so in my case the cookie is just another header.

My first thought was to simply trap for the Set-Cookie header and then parse out the cookie and create a Cookie object instead. But given that cookies can have a lot of different options this is not exactly trivial, plus I don't really want to fuck around with cookie values which can be notoriously brittle.

Don't use Response.ClearHeaders()

The real mystery in all this is why calling Response.ClearHeaders() prevents a cookie value later written with Response.AppendHeader() to fail. I fired up Reflector and took a quick look at System.Web and HttpResponse.ClearHeaders. There's all sorts of resetting going on but nothing that seems to indicate that headers should be removed later on in the request. The code in ClearHeaders() does access the HttpWorkerRequest, which is the low level interface directly into IIS, and so I suspect it's actually IIS that's stripping the headers and not ASP.NET, but it's hard to know. Somebody from Microsoft and the IIS team would have to comment on that.

In my application it's probably safe to simply skip ClearHeaders() in my handler. The ClearHeaders/ClearContent was mainly for safety but after reviewing my code there really should never be a reason that headers would be set prior to this method firing.

However, if for whatever reason headers do need to be cleared, it's easy enough to manually clear the headers out:

private void RemoveHeaders(HttpResponse response)
{List<string> headers = new List<string>();foreach (string header in response.Headers)
    {
        headers.Add(header);
    }foreach (string header in headers)
    {
        response.Headers.Remove(header);
    }

    response.Cookies.Clear();
}

Now I can replace the call the Response.ClearHeaders() and I don't get the funky side-effects from Response.ClearHeaders().

Summary

I realize this is a total edge case as this occurs only in HttpHandlers that are manually configured. It looks like you'll never run into this in any of the higher level ASP.NET frameworks or even in ASHX handlers - only web.config defined handlers - which is really, really odd. After all those frameworks use the same underlying ASP.NET architecture. Hopefully somebody from Microsoft has an idea what crazy dependency was triggered here to make this fail.

IAC, there are workarounds to this should you run into it, although I bet when you do run into it, it'll likely take a bit of time to find the problem or even this post in a search because it's not easily to correlate the problem to the solution. It's quite possible that more than cookies are affected by this behavior. Searching for a solution I read a few other accounts where headers like Referer were mysteriously disappearing, and it's possible that something similar is happening in those cases.

Again, extreme edge case, but I'm writing this up here as documentation for myself and possibly some others that might have run into this.

© Rick Strahl, West Wind Technologies, 2005-2012
Posted in ASP.NET   IIS7  

A Small Utility to Delete Files recursively by Date

$
0
0

It's funny, but for me the following seems to be a recurring theme: Every few months or years I end up with a host of files on my server that need pruning selectively and often under program control. Today I realized that my SQL Server logs on my server were really piling up and nearly ran my backup drive out of drive space. So occasionally I need to check on that server drive and clean out files.

Now with a bit of work this can be done with PowerShell or even a complicated DOS batch file, but heck, to me it's always easier to just create a small Console application that handles this sort of thing with a full command line parser and a few extra options, plus in the end I end up with code that I can actually modify and add features to as is invariably the case. No more searching for a script each time :-)

So for my typical copy needs the requirements are:

  • Need to recursively delete files
  • Need to be able to specify a filespec (ie. *.bak)
  • Be able to specify a cut off date before which to delete files
  • And it'd be nice to have an option to send files to the Recycle bin just in case for operator error :-)
    (and yes that came in handy as I blew away my entire database backup folder by accident - oops!)

The end result is a small Console file copy utility that I popped up on Github:

https://github.com/RickStrahl/DeleteFiles

The source code is up there along with the binary file you can just run.

Creating DeleteFiles

It's pretty easy to create a simple utility like DeleteFiles of course, so I'm not going to spend any talking about how it works. You can check it out in the repository or download and compile it. The nice thing about using a full programming language like C over something like PowerShell or batch file is that you can make short work of the recursive tree walking that's required to make this work.

There's very little code, but there's also a very small, self-contained command line parser in there that might be useful that can be plugged into any project - I've been using it quite a bit for just about any Console application I've been building.

If you're like me and don't have the patience or the persistence (that funky syntax requires some 'sticking with it' that I simply can't get over) to get into Powershell coding, having an executable file that I can just copy around or keep in my Utility directory is the only way I'll ever get to reuse this functionality without going on a wild search each time :-)

Anyway, hope some of you might find this useful.

© Rick Strahl, West Wind Technologies, 2005-2012
Posted in Windows  CSharp  
Viewing all 664 articles
Browse latest View live