Rui Figueiredo

Hack your brain to learn more, faster and better

May 4, 2016 by Rui Figueiredo Leave a Comment

I had a really hard time naming this post. It’s hard not to make it sound like something dodgy, like some of those ads that promise you 20K a week with some little secret that will be revealed if “you just click this link”.

No, this is not like that, it actually requires effort. This post is about being smart about that effort. It is mostly about memory, how it works, and how you can make it work in your favour.

There has never been a need to learn new things so quickly, especially if you work as a software developer. It feels like there’s a new framework every other day, new tools, new ways of doing things. Pick a name, add “js” to it, and chances are that you’ll find a library/framework/tool with that name. JavaScript fatigue is now a thing.

Turns out most of us don’t have a good understanding of how our memory works. We completely overestimate how long we will remember something that we have just learned. If we are exposed to something just once we tend to forget it really quickly. The graph below is known as the Forgetting Curve:

And the tragedy here is that the strategy that most people have for learning a new subject is to read, maybe practice a bit and then stop. And because we overestimate our ability to remember, when we actually need to remember we can’t.

To make matters worse there never seems to be enough time to study. We are expected to work long hours so we can only study a bit on the weekends, or every couple of days. It often happens that when we pick up where we left we forgot about what we learned from the previous time.

Turns out that not all is bad. If we use our memory properly we are able to do incredible things. Would you believe if I told you that with very little practice it is possible to memorise the order of a full deck of cards. There are even tournaments for that. The world record is 20.44s, that’s right, less than 21 seconds.

The techniques for doing that involve using your imagination to mentally build stories that involve all types of perceptions (e.g. spacial, smell, taste, sound, touch).

It does not hurt to know about these techniques. They will help memorise pretty much anything. If you want to learn more about them I recommend Dominic O’Brien’s book “You Can Have an Amazing Memory”. Also, if you want to try memorising a deck of cards try my Windows Store App Memory Ace (also on Windows Mobile).

Even though these techniques are very helpful, on their own they don’t solve the problem that given a few days you’ll forget what you learned if you don’t review it.

The solution is to accept that reality. We forget things, so we must review them in order to not forget them. At first this may seem impractical, it takes too much time, too much effort.

But that is not necessarily true, I want to share with you a “system” for doing this.

Every time you want to learn a new subject (e.g. technology, tool, framework, library, …) you read about it whenever you have time. Be it in the half hour train ride in the morning to work, during your lunch time, it really does not matter. But every time you learn a new concept you transform it into a question/answer pair.

For example, imagine you’ve decided to learn AngularJs and you’ve just learnt about the ng-show directive, you could write this question/answer pair:

Question:

Imagine you have an angular controller with 
a property named isVisible (with a boolean value) 
How would you use that in a template?
________________________________________

Answer:

<div ng-show="isVisible"></div>

Source: https://docs.angularjs.org/guide/animations

Or for example if you’ve learned about the ng-src directive:

Question:

Imagine you have an angular scope with a 
json object named phone. phone has a imgScr 
property with a url to an image
Use it in a template to display the image of the phone

____________________________________

Answer:

<img ng-scr="{{phone.imgSrc}}"/>

If you use src="{{phone.imgScr}}" the browser will perform 
a request as soon as the img tag is read, and this is before 
angular had the chance to replace it with the intended value.

Source: https://docs.angularjs.org/tutorial/step_06

After creating these question/answer pairs (which on their own help memorise whatever you are learning) you will then use them as your material for reviews. When you are reviewing, you read the question and try to answer it mentally.

The good news is, you don’t have to review very often. And the better news is that every time you review something the likelihood of you remembering it for longer increases (a lot). Here’s a graph that shows how that looks:

The optimal rate of review is relatively well known, and thankfully there is software that can schedule it for us. That software is often referred to as Spaced Repetition software.

My favourite software for doing this is Anki. When you use it, you add question/answer pairs and then Anki will schedule them for you. When you answer a question you can report how easy it was to remember the answer, and that will be used to schedule when to ask that question again.

I’ve been using it for a little less than three years (1013 days today). I can share some numbers with you so that you can have a notion of how much time the review process takes.

In these almost three years I’ve spent a total of 137 hours reviewing things. I spend an average of 8.7 minutes a day reviewing (I usually do it while I take breakfast). If you think about it that is an incredibly small amount of time given that I have added almost 3000 questions to Anki. That’s 3000 pieces of information that take an average of 8.7 minutes a day to maintain. And that number would certainly be lower if I wasn’t always adding more information (you have to review more often when the information is “fresh”).

If you are a gamer, you’ll probably immediately recognize how small a number 137 hours is. Nowadays games keep track of how many hours you’ve been playing them. I’m not a heavy gamer at all, only play a little bit online during weekends. I have more hours of playing Battlefield 4 than that.

Here’s how successful I’ve been in recalling:

Anki distinguishes the recency of the questions as “Learning”, “Young” and “Mature”. A mature question is over 21 days old; young is less than 21 days and learning is a question that has yet to be reviewed.

Practical tips for creating your own questions/answers

The most important tip is: do not use other people’s questions. The exercise of creating your own questions helps immensely to process the new information you are learning. Also, if you misunderstood something, that becomes clear very quickly when you are thinking about a way to transform that new knowledge into a question/answer pair.

Have questions that are focused. For example, do not write a question that asks you to enumerate all the options available in jQuery validation’s validate method. Add a question with a scenario relevant for each option if you really want to know them all by heart.

You need to understand the answer. If you don’t understand the answer you will always forget it.

What I mean by not understanding is: you can’t mentally link the information you want to remember with other things you know. For example, if you try to memorise that in angular calling .service is just syntactic sugar to calling .factory and returning a “service”, but you don’t know what a service and a factory are you’ll very quickly forget this fact.

This also helps you guide your study. Every time you find something you don’t know, you investigate that, and then come back to what you were studying. In this example, that would be figuring out what a factory and service are in angular.

This last tip might not seem too important, but I cannot stress enough how important it is. The sad fact is, most tutorials, books, manuals, training videos are not very good. A good technical writer is capable of having a mental model of what the reader knows given what was explained, however most seem to lack this skill. A lot of material will mention new concepts as if the reader is as familiar with them as the writer before they were properly explained in the text/video. The advice here is: stop, figure out what you don’t know, learn that, and then continue.

Conclusion

Accept that we forget things if we don’t review them. Reading a whole book to learn that new framework that will land a higher paying job during a weekend is not a good strategy. Instead while you read create review material (question/answer pairs) and review it every day (it takes less than 10 minutes per day).

This way you can use all those small free moments (train ride to work, lunch hour, a pomodoro in the evenings) because you will remember everything you learn. You can study a bit today and then continue a month from now if for some reason you could not do it before.

What exactly is a "unit" in unit testing?

April 27, 2016 by Rui Figueiredo 3 Comments

This should be an easy question to answer, right? Turns out that there are several definitions of unit testing, and even though they are similar, they are also different enough that answering this question is difficult.

Let’s look at some examples. Form Wikipedia:

In computer programming, unit testing is a software testing method by which individual units of source code, sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures, are tested to determine whether they are fit for use. Intuitively, one can view a unit as the smallest testable part of an application. In procedural programming, a unit could be an entire module, but it is more commonly an individual function or procedure. In object-oriented programming, a unit is often an entire interface, such as a class, but could be an individual method.

So, according to Wikipedia a unit can be “an entire module”, “an individual function”, “an entire interface/class”, but also “an individual method”.

From MSDN:

The primary goal of unit testing is to take the smallest piece of testable software in the application, isolate it from the remainder of the code, and determine whether it behaves exactly as you expect…

A little less vague, a unit here is described as the “smallest piece of testable software in the application”. This still leaves room for debate.

Martin Fowler acknowledges that unit testing (and consequently a unit) is not tightly defined, however finds these common elements in definitions:

…Firstly there is a notion that unit tests are low-level, focusing on a small part of the software system. Secondly unit tests are usually written these days by the programmers themselves using their regular tools – the only difference being the use of some sort of unit testing framework. Thirdly unit tests are expected to be significantly faster than other kinds of tests.

One of my favourite definitions is from Roy Osherove’s The Art of Unit Testing book where Unit testing is defined as:

A unit test is an automated piece of code that invokes the unit of work being tested, and then checks some assumptions about a single end result of that unit. A unit test is almost always written using a unit testing framework. It can be written easily and runs quickly. It’s trustworthy, readable, and maintainable. It’s consistent in its results as long as production code hasn’t changed.

There’s also the definition of “unit of work”:

A unit of work is the sum of actions that take place between the invocation of a public method in the system and a single noticeable end result by a test of that system.

That end result is the method returning a value, some public state of the class changing or the public method being tested invoking another method in a dependency.

I really like this definition, however, I find the choice of using the term “unit of work” unnecessary, especially because it is a loaded term. For example, when discussing Object Relations Mappers (ORMs) the term “unit of work” has a completely different (and well established) meaning that is misleading in this context.

The “unit”

Can we agree that when we are writing a unit test we call a single method on the class we are testing? If so, then why can’t we consider a unit to be the logical path (path that the execution of the code takes) through that method given our initial setup of the test.

The tests need to be deterministic, so for the a given setup the logical path will always be the same.

Let me show you an example using an Account class and a Withdraw method. If there are enough funds, the amount withdrawn is taken from the account’s balance, if not an InsuficientFundsException is thrown.

There are two possible logical paths through the Withdraw method, one where there are sufficient funds, and another where there aren’t. If we want to be thorough, we should write a test for each.

A little taste of test first development

Let’s have a look at how this definition works while we write the Deposit method for our account. And let’s do it test first.

For the first test we want to make sure that the balance gets updated when we make a deposit of a positive value, here’s the test (I’m using NUnit):

And here’s the implementation:

If the caller specifies a negative amount we want to throw an InvalidOperationException. Here’s the test for that:

And here’s the new implementation of the Deposit method:

We can see that up until now we have a test per logic path. We can add an extra test to cover the case where the amount is zero. However, I’d argue that that test exercises the same logical path through the code (unit) than the positive amount test (however, no harm will come in having that extra test).

If we want the method to behave differently when the amount is zero, then we’d have to add another if statement, and that would produce another possible logical path through the method, for which we could write another unit test.

You might be thinking these examples are super simple. How would something more complex, where there are other classes/dependencies involved fair? Does this notion of a unit being a logical path through the code hold?

Imagine that our Deposit method has this requirement of alerting the fraud department if the deposit looks suspicious. Let’s write our test for that, starting with a “suspicious” deposit (I’m using moq as my isolation framework):

And now the test for when the deposit isn’t suspicious:

And the new implementation of the Deposit method:

You’ve probably noticed that I added a few dependencies (IFraudDetectionService and INotificationService). We’ll get to that. But first, what happens to the tests we previously had?

We have to adjust them so that they take into account the new dependencies, but should we keep all of them? Maybe we can drop the Deposit_PostiveAmount_AdjustBalance test.

If we look at all the possible logical paths through the Deposit method we can see that the there’s no way that that test fails in isolation (path 2/3). If the balance is not updated correctly, both Deposit_SuspiciousAmount_UpdatesBalanceAndSendsEmailToFraudDeparment and Deposit_NonSuspiciousAmount_UpdatesBalanceAndDoesNotSendsEmailToFraudDeparment fail.

On the other hand, one can make the argument that we should have all tests, one for when the amount is negative, one that only asserts that the balance is updated correctly, another that verifies that the notification service is called when the deposit is suspicious, and another verifies that the notification service is not invoked when a deposit is not suspicious.

If we want that our definition of unit takes this into account, we can say that a unit is a verifiable change that occurs in a logical path through a method. That can be a change of internal state that is externally visible (e.g. the Account’s Balance), a method invocation in a dependency (e.g. verify that the notification service was called) or the return value of the method.

The advantage of doing this is that it is easier to figure out what is wrong just by looking at the name of the failing test.

One thing you might have noticed is that this idea of logical path is very similar to a software metric named Cyclomatic Complexity. A method’s cyclomatic complexity is “the number of linearly independent paths through the code”. In fact, right now, as I read the Wikipedia entry, I’ve just noticed this idea of using the logical path in testing is not new. That author of the Cyclomatic complexity metric (Thomas McCabe) describes this idea (one test per logical path) as a testing strategy in a paper from 1996 (way before unit testing was popular). It is named basis path testing.

The dependencies

This is a bit of an aside, but you should have noticed that I used interfaces so that I did not have to specify a specific criteria for when a deposit is considered suspicious, or how the “fraud department” is notified.

This not only made the code easier to test, it also has the advantage of isolating the Account class from changes in the criteria used to determine if a deposit is suspicious or how the notifications should be performed.

I brought this up just to mention that if you write the tests first you’ll have these problems in mind, you’ll want to write your code so that it is easy to test. You will feel compelled to write small focused methods, and be very mindful to what does and doesn’t belong in a class and should be considered a dependency.

Also, you should be in full control of the code that executes through the logical path you are testing. For example, in Deposit_SuspiciousAmount_UpdatesBalanceAndSendsEmailToFraudDeparment I used moq to create a stub of the IFraudDetectionService interface that always returns true when IsDepositSuspicious is called.

Say the implementation of that dependency made a call to a database or some other service over the network. That would make that test be considered an integration test. However, I’d argue that even if no database call was made, or any service called, you should still use the isolation framework and create the stub. It removes uncertainty. When the test fails and you are in full control of what was executed, you don’t have to guess what went wrong.

Principle of reasonable expectations

April 20, 2016 by Rui Figueiredo 1 Comment

The area of design has a very elegant principle named The principle of least astonishment. Sometimes it is also referred to as The principle of least surprise. It means that you should not do something that surprises the user, such as having a close button opening a new page, a trick that many spammy websites use.

This principle is very important in design because it helps guide the creation process. It guarantees that the design that is produced is easier for the user to use.

A great thing about this principle is that it is very easy to understand and we immediately can relate to situations where we felt it was violated.

There are principles in the area of software whose goal is also to guide the final product into a state where it has some desired properties, usually that it is easy to read and that it can easily withstand change.

Unfortunately software is more abstract than design. It is harder to visualise. Its principles are frequently misunderstood, and people tend to learn them from word of mouth. For example, the SOLID principles. Would you say most developers you know have read the original papers where they were introduced?

I, for example, have seen the single responsibility principle being invoked for the most bizarre things. One of the ones I’ve heard most often is for justifying putting everything that is remotely related in a single class, almost the opposite of what the principle means.

Maybe we need simpler principles.

Not that we don’t need the ones we already have, but maybe we can look at other areas and adapt their principles to software, especially the ones that we can visualize and relate to. These might be especially useful for novice programmers because they are easy to understand.

The principle of least surprise is one of them surely, and often people already use it in the context of software design.

Another principle that might be useful (and this one as far as I know hasn’t been used in software) is the principle of reasonable expectations.

It is a legal principle in that a contract should be interpreted as how a reasonable person (who is not trained in the law) would interpret it. It favours the objectively reasonable expectations of the weaker party (the reasonable person) even when the language of the contract does not explicitly supports them.

This principle emphasizes the “consumer” of the contract, in software we could draw a parallel with the person who has to maintain or use the code. And this person is not necessarily a different person than who wrote it. Given enough time, even the code that we wrote becomes as alien as anybody else’s.

The principle says that we should not violate reasonable expectations. Here are some examples:

In older versions of jQuery UI (1.8 for example) if you would check if a dialog was open, and it wasn’t, the method would return the DOM object that was queried instead of false, i.e.:

var isOpen = $('#theDialogContainer').dialog("isOpen");
if (!isOpen) //isOpen is a DOM object
    $('#theDialogContainer').dialog();

If this was the click handler for a button, it would never open the dialog because of how JavaScript evaluates conditions. It uses falsy and truthy values, and a DOM object is a “truthy” value.

One could almost argue that the concept of falsy and truthy values is a violation of the principle of reasonable expectations, however, because it is common knowledge for JavaScript developers it falls into a grey area. Nevertheless, no one would lose if the snippet above was rewritten this way:

var isOpen = $('#theDialogContainer').dialog("isOpen");
if (isOpen === false) {
    $('#theDialogContainer').dialog();

Other examples include things that the method does that it is not reasonable for someone to expect, for example, imagine you are using an API to control grids of elements (imagine a webpage with a dynamic grid/table of results). If this function is called: grid.select(element);, the row that contains the element gets highlighted and the page scrolls to it. The second part, the page scrolling to the row, is not something that a consumer of the API would expect just by reading the method name.

Let me just give you an example of how this can become problematic.

Say that you want to implement filtering. Every time the user filters the results, the previously selected element can be filtered out. If this happens you want the first element of the grid to become selected. It would be more than reasonable to use the select function to achieve this.

Up to this point, even after adding filtering it is unlikely that anyone using the select function would notice that it not only highlights the row where the element is at, it also causes the page to scroll to that element. This is because the filters are on top of the page, when they are used the first element is also visible, so the scroll function has no visible effect.

Time passes and a new requirement arrives for having keyboard shortcuts for commonly used filters. Now the user can cause the results to be filtered anywhere from the page. If the user scrolls to the bottom of the page and uses the shortcut to filter the results, the page sometimes (when the selected element is filtered out) jumps to the top.

The person tasked with resolving this bug will likely look for it first in the code that handles the keyboard shortcut that triggers the filtering. When that fails, it is also likely that eventually when the code that handles the filtering gets looked at, the select(element) method is not considered to be the culprit. This is because it is not reasonable to expect (unless the person solving the bug is familiar with the implementation) that selecting a element causes the page to scroll to the row that contains that element.

This problem could be solved by separating the function into two other functions and name them in a way that what they do is clear, for example, grid.highlight(element) and grid.scrollTo(element).

This example nicely illustrates why I think this principle has a place even though it may seem similar to the principle of least astonishment. It is easy to imagine whoever wrote the grid.select function to argue that it is not a surprise for the page to scroll to the row that contains the element when that element is selected. However, it is not reasonable to assume that someone who is not familiar with the implementation of the select method to expect it to have that behaviour.

Another useful thing that this principle surfaces is that if something provides no reasonable expectations, then it is a violation of the principle. Consequently, naming functions as for example handleTriggerFailed will violate the principle since there is not much that the reader of the code can infer from that method name. Other examples that apply to class names: AbstractService, SomethingManager, SomethingHelper, etc.

The principle of reasonable expectation puts the focus on the interpretation that the reader of the code is likely to have. It underlines the fact that the code’s main target audience is first the people and then the compiler. If that wasn’t the case we would all still be writing assembly.

Why composition is superior to inheritance as a way of sharing code

April 13, 2016 by Rui Figueiredo 5 Comments

Inheritance is one of the pillars of Object-oriented programming (OOP), the other two being polymorphism and encapsulation. Inheritance is therefore centre stage whenever OOP is discussed. There are several flavours of inheritance however. There is implementation inheritance and interface inheritance. It is the former that is the focus of this post.

Implementation inheritance is described as a way to achieve code reuse. This works by defining methods and variables in a class that is then inherited by other classes. Those other classes are said to be derived from the original class, and the original class is said to be their base class.

The derived classes have access to the methods and variables from the base class, and this is how code reuse is achieved.

The idea of sharing code this way dates back to 1967, specifically to the SIMULA language which is credited to have been responsible for the birth of object oriented programming.

This concept dates back almost 50 years, it must be a great idea because it is still being used today. But put that thought aside and consider this instead. How would the scale of the programs written at that time compare with what we do today? I wasn’t born at that time, but I’ve heard from people who lived it that 500 lines of code was a very large program by the standards of those days. Today, that is very little.

Also, artificial intelligence was doing well at that time (this was before the AI winter), and there was this notion of IS-A that was very popular. IS-A is a concept that came from the field of Knowledge Reasoning, namely Semantic Nets. In a semantic net there are nodes and arcs. A node represents an entity, and an arc represents a relationship between entities. One type of arc is named IS-A where the properties from the source node of the IS-A relationship are inherited by the target node. Sounds familiar?

This idea had to come from somewhere, and although it is very useful in the field of Knowledge Representation and Reasoning where you can perform inference based on it (e.g. a Cat is a mammal, all mammals have fur, so a cat has fur), it is not so useful in OOP. In fact, the best ever description of this concept when applied to OOP that I’ve heard was from Uncle Bob: “…inheritance is not ISA. Inheritance, if you look at it with a very jaded eye, inheritance is the declaration of methods and variables in a subscope and it has nothing to do with ISA…”

This idea of IS-A is still popular today when describing how to use inheritance. Just do a search for “object oriented programming inheritance” and select images.

So why is it that implementation inheritance is not a good tool for code reuse? Well, it is heavily based on ideas from another area that do not apply in OOP, and there are better alternatives. I’m speaking of course, of composition.

First, let’s look at how code reuse looks like through inheritance and then how it looks like through composition.

public class Base 
{
    public void SharedMethod(){
        //do something
    }   
}


public class MyClass : Base
{
    public void Method()
    {
        //SharedMethod is available here
    }   
}

The first obvious thing is that the code we want to reuse is only available to derived classes, also most modern languages only allow one base class per class.

This design is opaque. What I mean by this is that by looking at the derived class the only information you have about the shared method is its name. Frequently you need to traverse the hierarchy up to understand what is going on.

Moreover, if the derived class has more than a couple of lines of code it becomes hard to distinguish which methods are defined in the base class and which are defined in the derived class, because there’s nothing to tell them apart.

And finally, it is not a test friendly design. Imagine that the SharedMethod makes a call to the database and you want to write unit tests for the MyClass class. It’s a hassle to mock the result of a call to the SharedMethod.

How will this look like using composition?

public interface IDependency 
{
    void SharedMethod();
}

public class Depedency : IDependency 
{
    public void SharedMethod()
    {
        //a shared method's implemetation
    }
}

public class MyClass    
{
    private readonly IDependency _dependency;

    public MyClass(IDependency dependency)
    {
        _dependency = dependency;
    }

    public void Method()
    {
        _dependency.SharedMethod();
    }   
}

Now any class can use the code that we want to reuse, they just need to reference the dependency that has the code we want to share.

There’s more information in MyClass for us humans. It’s not just a call to the method we are reusing without any context. Replace IDependency by IMailService and shared method by SendEmail and it becomes obvious what it does and where it is implemented.

You’ve probably noticed that I used an interface IDependency instead of a concrete class. This is just another thing that composition enables that is not possible when sharing code through implementation inheritance. When you use implementation inheritance you tie your derived classes to the particular implementation in the base class.

If you want to write unit tests for MyClass then it is very easy to use any isolation framework to be able to mock the calls to IDependency and test the method of MyClass in isolation.

I almost feel like calling implementation inheritance automated copy & pasting, which if you think about, in its purest form, that’s what it is.

How come it is so popular, even though it’s a less flexible way of sharing code? The way it is taught is very appealing, especially the IS-A concept seems to make a lot of sense at first. I think this is why it prevails as probably the first thing most developers think when they want to share code.

However, when you take a hard look at implementation inheritance as a tool to share code you can quickly find that it is inferior, and that is why probably there are so many sources saying that you should prefer composition over inheritance. This is another one, with a little bit of history.

7 tips to find code faster in a web application

April 6, 2016 by Rui Figueiredo Leave a Comment

It’s difficult to locate where a certain functionality is implemented in a code base that is unfamiliar to us, and even if you are familiar with the code base, when a project grows in size (imagine 3+ years), it inevitably becomes harder to manage. Locating code takes more time, and it stops being obvious where a certain piece of functionality is implemented.

In this blog post I’ll share with you a set of tips and tricks to help you go from web page, to JavaScript code, to server side code that is responsible for rendering that page.

For the server side I’ll focus on ASP.NET MVC, which is my cup of tea. If you use any other framework I still believe they’re applicable, and if you know how to realise them in your framework of choice please feel free to chime in the comments.

Tip 1. Starting point: The Url

The Url gives a lot of information, and it is a good starting point. For example, if your web app is build using ASP.NET MVC you could take from a Url like this: http://blinkingcaret.com/Account/Login that the code that renders the page is in a Controller named AccountController, and an action (method) named Login.

The default view name in this scenario would be Login.cshtml, the same name as the action method, however this can be overridden, and the view name and it’s location can actually be arbitrary.

Speaking of defaults, even though the default behaviour in this scenario is that the controller and action name are Account and Login, respectively, this can also be overridden. MVC frameworks use the concept of a router, which is a component responsible extracting from the Url which controller and action to invoke. It is possible to specify a route other than the default one (and that route can be “Account/”) and specify an arbitrary default controller.

So, even though the Url is a good starting point, most frameworks provide mechanisms to override the default behaviour of how a Url is ultimately translated into the code that renders the view. Tip 7 provides an alternative that, at least for ASP.NET MVC, will give you the right controller and view every time (it might still be applicable to other frameworks if they have similar extensibility points).

Tip 2: Chrome developer tool’s inspector (Elements tab) for HTML/CSS

Chrome developer tools offer excellent help in finding code. They are accessible by pressing F12 (or Control+Shift+I for those used to use Firefox’s Firebug).

One straightforward way to take advantage of the dev tools to find where the code that generates the markup is located is to just inspect an element, find the snippet of markup you are interested in and then use Find in Files functionality of the IDE. I wouldn’t recommend this approach too often though. On big projects the search tends to take a reasonable amount of time, and we all know that more than 5 seconds is enough to create the temptation to open a new tab and go read random stuff on the internet. Again, tip 7, where applicable, is a better alternative.

Tip 3: Markup injected in the page via Ajax

This tip is for when there is content being injected in the page via Ajax, and you want to quickly find where is the JavaScript code responsible for it.

You can set chrome dev tools to break on “Sub tree modifications”. If you do this Chrome will break every time anything gets added or removed from the DOM and then you can follow the callstack until you find the code responsible for injecting the html in the page.

Tip 4: Plain JavaScript events

It is often the case that we need to find what JavaScript is running in response to a user’s action, for example, the click of a button, or a change in a dropdown.

If the event was registered using plain JavaScript (i.e. not with jQuery), chrome dev tools has a tab, named Event Listeners, that list all the event handlers registered for the element that is selected.

When you are using jQuery to register events, these won’t show up in the Event Listeners tab though. It is jQuery itself that handles these events. That leads us to:

Tip 5: jQuery events

If an event was registered using jQuery and we want to check which code is running in response to it we’ll have to use jQuery’s api.

The data regarding which events are registered for which element is accessible by using the $._data method. For example, if you want to check which jQuery events are registered in the element with id="myElement" you could write this in the console:

$._data($('#myElement').get(0), "events")

Or just take advantage of the fact that in chrome dev tools the element you select in the elements tab is accessible in the console by writing $0.

Every time you get chrome to show you a function handler in the console you can right click on it and select Show function definition and that will take you to the code that runs in. The code will open in a new tab with the same name as the file name that contains the code. You can even add breakpoints and step through the JavaScript code if required.

In jQuery there’s the notion of delegated events, where you define an event handler in a parent element and you specify a selector for the child elements. When an event is raised on a child element that matches the selector on the parent, the event handler runs, for example:

$('div#container').on("click", ":button", function(e) { console.log(e.target.id + " was clicked");});

This event handler will run every time a button is clicked inside div with id=”container”. This is very useful if the contents of the div are injected via ajax. You don’t have to register handlers for the elements inside the div this way. This has the disadvantage of making finding those handlers hard.

I’ve made a tool named findHandlersJs that will find all the jQuery event handlers that run in an element, it’s syntax is very simple findEventHandler(eventName, element(s)), for example:

findEventHandlers("click", "div#container :button")

or if you have the element selected in the Elements tab in chrome, you can simply write:

findEventHandlers("click", $0)

It returns information about not only any event handler registered in the buttons inside div with id=”container” but also any delegated event handlers that run in response to a click in that button(s).

Tip 6: Client side frameworks (KnockoutJS and AngularJS)

It is possible to access a lot of information from the console even if you are using client-side frameworks such as angular or knockout to generate your client side markup.

The examples here are only for Angular and Knockout because these are the ones that I’m more familiar with, I’m sure there are equivalent methods of getting to this information with other frameworks. If you have some tips for other frameworks, feel free to share in the comments.

For KnockoutJS it is possible to get to the ViewModel, and from there you have access to the functions that run in response to the user’s actions. For example if you want to debug the code that runs in response to the user clicking a button:

You can use the ko.dataFor(domElement) to get the ViewModel that was used. In the ViewModel you can not only access the observables, but also all the functions that are invoked in response to users’ actions. Alternatively, you can use ko.contextFor to access the specific binding context of the element.

For AngularJS, it is similar, you use the angular.element(domElement).scope() to access the angular scope for the element. From the scope you can then go to the functions that are invoked in response to the user’s actions.

Tip 7: Leverage the extensibility points of the Framework that you are using to inject debug information when rendering the pages

Even though this example is for ASP.NET MVC, this should be applicable to any framework that has similar extensibility points.

ASP.NET MVC renders the views through a view engine, and the framework has extensibility points for new view engines to be added, in fact, out of the box it comes with two view engines, one for WebForms (.aspx) and another for Razor views (.cshtml).

I’m going to show you how you can create your own version of the Razor View Engine that adds extra debug information as html comments to the pages that are rendered.

Comments are a better solution, than say for example, wrapping a partial view inside a div with attributes because doing so could potentially interfere with css rules. Html comments are innocuous, and are rendered in a different colour in chrome dev tools which makes them easy to spot.

First you’ll have to create a new view engine class, for example RazorDebugViewEngine, that inherits from RazorViewEngine.

Next override the method that creates the (full) views (CreateView) or the one that creates partial views (CreatePartialView), or both.

We’ll override both in this example and will provide different comments for each.

Usually RazorViewEngine returns an instance of RazorView from the Create methods. We’ll provide our own RazorDebugView that we will define shortly:

public class RazorDebugViewEngine : RazorViewEngine
{
    protected override IView CreateView(ControllerContext controllerContext, string viewPath, string masterPath)
    {
        return new RazorDebugView(controllerContext, viewPath,
                     layoutPath: masterPath, runViewStartPages: true, viewStartFileExtensions: FileExtensions, viewPageActivator: ViewPageActivator);
    }

    protected override IView CreatePartialView(ControllerContext controllerContext, string partialPath)
    {
        return new RazorDebugView(controllerContext, partialPath,
                            layoutPath: null, runViewStartPages: false, viewStartFileExtensions: FileExtensions, viewPageActivator: ViewPageActivator)
        {
            IsPartialView = true
        };
    }        
}

I had to check the source code for MVC to figure out how the RazorViews were being created, it’s really great that this is open source :).

The IView interface (that RazorView implements) defines the contract for the Render method (void Render(ViewContext viewContext, TextWriter writer);) which is what ultimately renders the view. We are going to hijack that process by having the view render to a StringWriter instead so that we can access the generated html as a string.

We’ll add extra information, available from the ViewContext, to the html that was generated. Then we let the process progress as it would have otherwise by writing the generate content plus the debug information to the original TextWriter.

We’ve also added a property named IsPartialView to our RazorDebugView so that we can easily distinguish from a partial or full view.

Here’s how that looks like:

public class RazorDebugView : RazorView
{
    public bool IsPartialView { get; set; }

    public RazorDebugView(ControllerContext controllerContext, string viewPath, string layoutPath, bool runViewStartPages, IEnumerable<string> viewStartFileExtensions) 
        : base(controllerContext, viewPath, layoutPath, runViewStartPages, viewStartFileExtensions)
    {
    }

    public RazorDebugView(ControllerContext controllerContext, string viewPath, string layoutPath, bool runViewStartPages, IEnumerable<string> viewStartFileExtensions, IViewPageActivator viewPageActivator) 
        : base(controllerContext, viewPath, layoutPath, runViewStartPages, viewStartFileExtensions, viewPageActivator)
    {
    }

    public override void Render(ViewContext viewContext, TextWriter writer)
    {
        StringWriter stringWriter = new StringWriter();
        base.Render(viewContext, stringWriter);
        var renderedHtml = stringWriter.GetStringBuilder().ToString();


        string startDebugInfo;
        string endDebugInfo;
        if (IsPartialView) {
            var debugText = string.Empty;
            if (viewContext.IsChildAction)
                debugText = string.Format("ChildAction: controllerName: {0}, actionName: {1}, viewPath: {2}", viewContext.Controller.GetType(), viewContext.RouteData.Values["action"], ViewPath);
            else
                debugText = string.Format("viewPath: {0}", ViewPath);

            startDebugInfo = string.Format("\n<!-- BeginPartialView {0}  -->\n", debugText);
            endDebugInfo = "\n<!-- EndPartialView -->\n";
        }
        else
        {                
            startDebugInfo = string.Format("\n<!-- BeginView controllerName: {0}, actionName: {1}, viewPath: {2} -->\n", viewContext.Controller.GetType(), viewContext.RouteData.Values["action"],ViewPath);
            endDebugInfo = "\n<!-- EndView -->\n";
        }

        writer.Write(startDebugInfo + renderedHtml + endDebugInfo);
    }       
}

To use this new ViewEngine we have to add it. The best place to do it is in Global.asax. You can use the #if compiler directive so that you only use this view engine in debug mode. That would look like this:

public class MvcApplication : System.Web.HttpApplication
{
    protected void Application_Start()
    {
      //...

    #if DEBUG
        ViewEngines.Engines.Clear();
        ViewEngines.Engines.Add(new RazorDebugViewEngine());
    #endif
    }
}

Here’s a page using this new view engine would look like in chrome dev tools:

That’s it. Hope these tips are useful. If you have more please share in the comments.

Time to start getting familiar with Owin/Katana

March 30, 2016 by Rui Figueiredo Leave a Comment

Owin showed up sometime in 2012. The first time I’ve heard about it was on an Hanselminutes episode about Kayak (a web server build in entirely using .net). It is already used in ASP.NET 4 and it will be heavily used in ASP.NET Core.

The idea behind Owin is to provide an abstraction that both web servers and frameworks can use. It boils down to two important concepts, the environment dictionary, and the application delegate.

The environment dictionary is a collection of key value pairs that describe a request and the response to that request. In .Net this is literally a IDictionary<string, object> with keys such as owin.RequestBody, owin.RequestHeaders or owin.ResponseBody.

The application delegate is a function that takes an environment dictionary and returns a Task (Func<IDictionary<string, object>, Task>).

Picture a web server that can take an http request, create an environment dictionary with the data from that http request, pass it on to an application delegate, wait for the delegate’s task to finish and finally, create a response using the data in the environment dictionary (that was updated by the application delegate) and send it.

If you think of the application delegate as ASP.NET MVC (assuming that the environment dictionary is converted to a Request object, and the Response that MVC produces can be incorporated in the environment dictionary), it becomes clear that MVC could be used in any server that is capable of producing an environment dictionary. This also means that any web framework that can take an environment dictionary can be used in any server.

If you think about it, the environment dictionary is the minimum that is required to describe a request and a response to that request. The application delegate is the minimum that is required to specify a “handler” for that request.

Owin Middleware

Even though the definition of application delegate seems very simple, “it is just a function that takes a dictionary as a parameter and returns a Task“, the way it is used in the Owin specification is nuanced.

At first glance the application delegate might seem like it abstracts a framework, such as ASP.NET MVC. In reality it represents the execution of several components that form a pipeline to “inspect, route, or modify request and response messages”. Under this perspective, ASP.NET MVC is just one of those components, and the application delegate represents executing them all.

Each of these components is called an Owin Middleware. In .Net’s implementation of Owin, named Katana, a middleware takes the form of a class that in it’s constructor has a parameter for the next middleware to call. This is how the pipeline is formed.

In Katana’s implementation of Owin the middleware runs in a particular order. Say, for example that you have three middlewares:

Middleware1

Middleware2

Middleware3

They will run in this order Middleware1, Middleware2, Middleware3, Middleware2, Middleware1.

This is easier to understand if we look at what is actually called when a middleware runs:

public async Task Invoke(IOwinContext context) {
    //do some work
    await Next.Invoke(); //this will be the rest of the pipeline running
    //do some more work
}

(IOwinContext is how the environment dictionary is accessed in Katana. Think of it as a friendlier version of the environment dictionary)

A real world example will make this even clearer. In an ASP.NET MVC5 project created with Individual User Accounts you can find, by looking in the Startup class (and other code that is called from there), these middlewares being defined:

public void Configure(IAppBuilder app)
{
    app.UseCookieAuthentication(...)        

    //...       

    //app.UseMvc();
}

(In Katana (the .Net implementation of Owin) this is how you build the pipeline: using the Startup class and IAppBuilder).

Every time a request comes through this pipeline it will first be handled by the cookie authentication middleware. The first thing that this middleware will do is check if the request contains a valid authentication cookie. If so, it creates an IPrincipal, that represents the user, using the data in the cookie, and then it invokes the next middleware. If not, it just invokes the next middleware.

The next middleware is ASP.NET MVC (it is “commented out” in the example because in this version of MVC this is done implicitly, however in MVC6 this is will be done explicitly).

If there was no cookie in the request (and therefore the user is not signed in) and the resource the user is trying to accept requires a signed in user, ASP.NET MVC will return a response with status code 401 Unauthorized. After this, the cookie middleware will run again, inspect the environment dictionary. It will be looking for a response code of 401, which it will find in this case. It will then change that response to 302 Redirect to the login page.

In very broad strokes this is how authorization works now. Previously this work was done by an HttpModule and an HttpHandler. The module was FormsAuthentication and the handler was MVC’s IHttpHandler implementation.

Owin/Katana and ASP.NET Core, MVC6 and MVC5

If you create an ASP.NET MVC6 project you’ll notice that it has a Startup.cs class, you’ll also notice that there isn’t a web.config. There are no http modules and handlers. This seems to be the way forward.

But don’t worry if you feel that you are still far away from using MVC6. You can use Owin/Katana today in you MVC5 projects. There’s a Nuget package that you can install in your MVC5 project, Microsoft.Owin.Host.SystemWeb that will do the trick of converting requests to the environment dictionary (I had to ask how this works, it wasn’t/isn’t documented anywhere).

You then have to add a Startup class, there’s a visual studio template for that, just right click on the web project -> add -> owin startup class. You can now start using owin middlewares.

There’s a lot more to this, so I’ll leave you with some resources so that you can explore more about Owin and Katana:

http://odetocode.com/blogs/scott/archive/2013/07/09/getting-started-with-owin-katana-and-vs2013.aspx

http://odetocode.com/blogs/scott/archive/2013/11/11/writing-owin-middleware.aspx

http://odetocode.com/blogs/scott/archive/2013/11/12/simple-logging-middleware-katana-part-4.aspx

https://blogs.msdn.microsoft.com/webdev/2014/11/14/katana-asp-net-5-and-bridging-the-gap

http://www.asp.net/aspnet/overview/owin-and-katana/an-overview-of-project-katana

http://www.asp.net/aspnet/overview/owin-and-katana/owin-startup-class-detection

http://docs.asp.net/en/latest/fundamentals/owin.html

http://docs.asp.net/en/latest/fundamentals/middleware.html

How to manually use the ASP.NET MVC’s client side validation infrastructure

March 23, 2016 by Rui Figueiredo 2 Comments

ASP.NET MVC client side validation is based on the jQuery validation plugin. It can be said that MVC’s client-side validation is an opinionated version of how jQuery validation should work in an ASP.NET MVC project. Despite this, the underlying implementation is fully based on jQuery’s. In this blog post I’ll show you how you can take advantage of this.

ASP.NET MVC Client Side validation requirements

First, here’s the list of things you need to do to enable client-side validation in an ASP.NET MVC project Make sure your client side code is loading both:

jquery.validate.js
jquery.validate.unobtrusive.js

Make sure your web.config has the following keys in appSettings with the follwoing values:

  <appSettings>
    ...
    <add key="ClientValidationEnabled" value="true" />
    <add key="UnobtrusiveJavaScriptEnabled" value="true" />
  </appSettings>

These settings can be overridden in a controller, make sure that is not happening. For example this would turn off client side validation if executed inside a controller’s action:

public ActionResult Index()
{
    HtmlHelper.ClientValidationEnabled = false;    
    return View();
}

The next requirement is that you use attributes from System.ComponentModel.DataAnnotations in the Model class that is used in view where you want client-side validation enabled.

For example, if we want the Email field to be a valid email, and make the password and email fields required we would create a model like this:

using System.ComponentModel.DataAnnotations; 

public class LoginViewModel
{
    [Required]        
    [EmailAddress]
    public string Email { get; set; }

    [Required]                
    public string Password { get; set; }            
}

Finally, we have to use the HtmlHelpers that generate the correct markup for all of this to work, and they have to be inside a form, for example

@model LoginViewModel

<form>
    @Html.LabelFor(m => m.Email)
    @Html.TextBoxFor(m => m.Email)
    @Html.ValidationMessageFor(m => m.Email)

    @Html.LabelFor(m => m.Password)
    @Html.PasswordFor(m => m.Password)
    @Html.ValidationMessageFor(m => m.Password)
</form>

Getting away with using Client Side validation without a model

The last two requirements are actually optional. It is possible to take advantage of client side validation without having to create a model class and annotate it, which can be useful if you only use a couple of parameters (such as in the Login example).

If you inspect the markup that the helpers generate you’ll see that it’s actually pretty simple:

<input type="text" name="email" data-val="true" data-val-required="The email is required"/>
<span class="field-validation-valid" data-valmsg-for="email" data-valmsg-replace="true"></span>

It turns out that to enable client side validation without using the HtmlHelpers and a model you just have to add an input with data-val="true" and then data-val- followed by validation method that you want to apply (e.g. data-val-required), the value of which will be the error message presented to the user (e.g. data-val-required="This is the error message"). This works because the MVC’s “unobtrusive validation” works by looking for inputs that are annotated with data-val attributes.

The data-valmsg-for‘s value is the name (not the id) of the input it refers to, and data-valmsg-replace="true" just means that the default message should be replaced, for example you could have a default message for the email field:

<span class="field-validation-valid" data-valmsg-for="email" data-valmsg-replace="true">The email must be unique</span>

This message would then be replaced by any validation error that occurs in the email field, for example “The email is required”. If data-valmsg-replace="false" then the original message will never be replaced. The only consequence of an error is that the span’s class is changed from field-validation-valid to field-validation-error (this happens irrespectively of the value of data-valmsg-replace="false").

Some validation methods have parameters, for example RegularExpression. The way these work is very similar, they just need additional data-val- for their parameters. If you want to validate a text field using a regular expression for 5 to 8 digits, it would look like this:

<input type="text" name="number" id="number" data-val="true" data-val-regex="Valid input is 5 to 8 digits" data-val-regex-pattern="^\d{5,8}$"/>

If you create the markup yourself you can get away without having to create a model for your view. Using the login example from above, your controller action for handling the user logging in could simply be:

    public ActionResult Login(string email, string password)
    {           
        ...
    }

You’d have to make any server-side checks on the parameters yourself though.

Here is the list of the System.ComponentModel.DataAnnotation attributes you can use, and their data-val counterparts:

Compare
- data-val-equalto="Error message"
- data-val-equalto-other="The name of the other field"
CreditCard
- data-val-creditcard="Error message"
EmailAddress
- data-val-email="Error message"
MaxLength
- data-val-maxlength="Error message"
- data-val-maxlength-max="Maximum length (e.g. 5)"
MinLength
- data-val-minlength="Error message"
- data-val-minlength-min="Minimum length (e.g. 2)"
Range
- data-val-range="Error message"
- data-val-range-max="Max value"
- data-val-range-min="Min value"
RegularExpression
- data-val-regex="Error message"
- data-val-regex-pattern="The regular expression (e.g. ^[a-z]+$)"
Required
- data-val-required="Error message"
StringLength
- data-val-length="Error message"
- data-val-length-max="Maximum number of characters"

There are also a few validation methods you can use that don’t seem to have a counterpart in System.ComponentModel.DataAnnotation. In fact you get a list of all the available client side validation methods by typing (for example in chrome) dev tools console: $.validator.unobtrusive.adapters. Here’s the list of the ones that don’t have a matching attribute: date, digits, number, url, length, remote, password.

Split your web application into smaller web applications

March 16, 2016 by Rui Figueiredo Leave a Comment

Working in big projects can be difficult, however most of that difficulty comes from accidental complexity (mostly unnecessary complexity introduced by ourselves). When a code base reaches a certain dimension this frequently happens:

Increase in the average time taken to locate where the functionality for a give feature is implemented
Low quality changes and hacks are more frequent. Sometimes this happens because people lose patience, sometimes they don’t know better. In a bigger code base it is easier for these types of changes to slip in, because it’s just harder to notice them.
Day to day ancillary coding tasks like checking code in, getting changes, waiting for integration tests to finish take longer.

Everything we can do to mitigate this is welcome.

If instead of looking at a big application as a big application, we instead look at it as a conglomerate of small applications working together, perhaps we can maintain the levels of productivity that are characteristic of green field projects, well beyond when a project reaches a respectable size.

Web applications are an ideal class of applications where we can apply this. The stateless disconnected nature of the web lends itself well to having small apps that work together but share very little code and ideally, no state between them.

When you are using a website today, outside of the request/response cycle, the server maintains no information about the client. A connection is open, an http request is sent to the server, the server sends back an http response. End of story.

The server knows about you, not because it keeps track of your IP address or any other similar mechanism. The server knows about you because when you sign in, the server gives you a cookie that contains information that only it understands (because it’s encrypted and signed, hopefully), and that uniquely identifies you. And on each subsequent request, you (your browser, really) sends it back, every time, until you sign out.

A cookie applies to a whole domain (e.g. blinkingcaret.com), meaning, each time you visit some page inside that domain (e.g. blinkingcaret.com/myawesomepage) the cookie is sent.

Nothing stops you from taking advantage of this to break your web application into smaller web applications, for example one that responds to yoursite.com/music, the other to yoursite.com/movies. They can be totally different apps, developed by different teams, different code bases. I promise you, if you use the same styles, no one will ever know.

For this to really kick in, try this, it will only take 5 minutes: open 2 instances of Visual Studio, on each do File -> New Project -> Asp.New Web Application -> MVC (set authentication to Individual User Accounts). After both visual studio instances finish creating the projects run them both at the same time. Register a user in one of them (that user will be logged in, since after registration you are automatically signed in). Now go to the other project’s tab and refresh the page. You are signed in on both! That other app doesn’t even have a database created yet. The default template uses the CreateIfNotExists database initializer, which didn’t even have a chance to run since you performed no action that involves an Entity Framework DbContext). Despite this, you’ll see your email/username at the top.

Example

Setting up a project this way involves several considerations. Namely:

Sharing layout pages and styles
Links between pages in different sub web applications
How to setup your development environment
Setup authentication and authorization in each of the web applications
Share components between different sub web applications

I’ve setup a sample project in GitHub using ASP.NET MVC. It’s an hypothetical store that sells music and movies. The main application is configured to deal with signing in users and then there are two other sub web applications, one for music, and another for movies. All the apps are setup to run in (Local) IIS, you’ll have to setup the main app to run on port 48000 using IIS Manager or else you’ll get an error while opening the project in Visual Studio, also you will need to make sure all projects are built before running the main web application. The reason for using Local IIS will become clear in (3). Also, all the projects are in the same solution, this was just more convenient for me. There’s no reason not to have several solutions.

The example project has no database. I initially created it with a database, but in the end decided that as an example, it was not worth the hassle of having to deal with all the necessary configuration just to run a sample.

Signing in an signing out users is therefore “faked” by manually creating a ClaimsIdentity and using Owin’s cookie middleware. If you are unfamiliar with this, check Security testing without committing to a particular authentication mechanism in ASP.NET MVC and What does it mean to be authenticated in ASP.NET MVC?.

1. Sharing layout pages and styles

The first hassle you’ll encounter when doing this is “how do I share the layout between each of the web applications?”. You might be tempted to use the Add Existing Item As a Link feature in visual studio for the _Layout.cshtml file. Unfortunately that is not supported for razor views. So the easiest thing to do is to have multiple _Layout.cshtml files, one in each sub web app.

It’s not ideal, this is definitely a compromise. Fortunately however, this is only necessary for your layout file. You can host all your style sheets (and some javascript if appropriate) from your main web application and access it from your sub web applications.

2. Links between pages in different sub web applications

This is another compromise. You won’t be able to use MVC’s mechanisms to generate these links. Url.Action, Url.RouteUrl, Html.ActionLink, etc won’t work. You’ll have to provide full links, for example, https://yourwebsite.com/musicstore/latestnews to access the latest music news from the main web application. Thankfully this should be rare enough that it is not a major maintenance problem.

3. How to setup your development environment (machine key, local IIS)

You should setup your web applications to run in a local instance of IIS, instead of running them in IIS Express (this is what happens when you create a new web project and run it in visual studio with the default configuration). Thankfully, this is a very simple change in the project’s configuration (you can do it in your projects properties and using IIS Manager). Also, you won’t be able to use LocalDb and you will instead have to use either full SQL Server, or SQL Server Express.

The reason why this is recommended is because you can setup the main application as a site in IIS and the sub applications as IIS applications within the site. This way they can all share the same port, i.e., they can all be located under http://localhost:48000, for example. This makes certain tasks easier, for example, you wont have to deal with CORS if you want to perform ajax calls between the web applications (even though they are all in the same domain, if they are hosted in different ports, it’s as if they are in different “origins” in CORS terminology).

Another important thing to be aware is that the Machine Key must be set to the same value in all web applications. The machine key is used to encrypt and decrypt the cookies. When you run in IIS Express this is not an issue, however, in local IIS you’ll have to manually set the keys (if you ever had a web application behind a load balancer you probably have done this already, each of the web applications has to be configured with the same key). The easiest way to do this is to use IIS Manager, double click machine key, uncheck both automatically generate at runtime, use the generate keys funcionallity, and the copy those keys (validation + decryption key) to all other sub web applications.

4. Setup authentication and authorization in each of the web applications

You should setup authentication in all the web applications, however, the login/register/forget password/etc pages should only be in one place. Typically the main application, and you should setup the sub web applications to redirect the user to the main login page in the main app if he tries to access a resource and is not authenticated/has insufficient permissions. If you are using a security token service, you also need to have the configuration in all apps.

In case you are using the Owin cookie authentication middleware you have to be aware that you’ll have to handle the redirects, since the cookie authentication middleware normally does not allow redirects to urls “outside” of the application it is running.

An example will make this clearer. Imagine your main application, where the login page lives, is in http://localhost:4800/ (the login page is in http://localhost:4800/account/login).

You have setup an action in the music controller to require the user to be authenticated. This action lives in a sub web app in http://localhost:4800/musicstore. So if your user types in the browser http://localhost:4800/musicstore/music/buy you want that user to be redirected to http://localhost:4800/account/login.

To achieve this, this is how you would setup your cookie authentication middleware in the musicstore sub web application’s Startup.cs:

app.UseCookieAuthentication(new CookieAuthenticationOptions
{
    AuthenticationMode = AuthenticationMode.Active,
    AuthenticationType = "ApplicationCookie",
    LoginPath = new PathString("/account/login"),
    Provider = new CookieAuthenticationProvider
    {
        OnApplyRedirect = context =>
        {                    
            context.Response.Redirect("http://localhost:48000/account/login");
        }
    }                
});

The LoginPath = new PathString("/account/login") is actually irrelevant (it would redirect to http://localhost:4800/musicstore/account/login), since we are taking over the redirect mechanism by providing our handler to OnApplyRedirect. This way we ensure that if the user is not authenticated he will always be redirected to the right login page, in the main web application.

5. Share components between different sub web applications

Apart from sharing razor files there’s no limitations about sharing “components”. Say for example that you have a controller action that returns the latest music news in the music web application and want to show those news in the main web application home page. You can just make an Ajax call to fetch the html for the latest news from the main application to the music web application, and inject that html in the home page. There’s no need to deal with CORS, and no other inconvenience apart than not being able to use the MVC helpers to generate the url.

Conclusion

Splitting up your web application into several web applications may seem like a lot of work right off the bat, but it’s something you will only have to do once.

Compare that to the pain of having to deal with a project with hundreds of controllers and views; waiting for updates and check ins/commits/fetches pulls and pushes; waiting for integration tests to finish that are unrelated to your changes on your build server; having to deploy the whole application just because of a small change in a specific section of the app; having someone else’s change, maybe incorrect IoC configuration, floor the whole web site, etc…

Even if you are a small team working in a small project, the benefits of being able to deploy parts of the web application separately might be enough to make it worth it. There’s only a small price to pay up front, and a few things you have to be aware while going forward. In the end, maintaining several small applications is always easier than maintaining a big one, and if those small applications work indistinguishably, then why not?

Setup ASP.NET Identity using StructureMap

March 9, 2016 by Rui Figueiredo 6 Comments

TL;DR: If you find yourself in a hurry, and you just want to quickly check how to setup ASP.NET Identity in a MVC project using StructureMap you can just check the example project in github

Although ASP.NET project templates are a very useful way to start a new project, some people don’t feel comfortable using them. There are many reasons for this, some people don’t like the project folder structure, others simply like to create the whole project themselves instead of starting from a sample application.

This was at some point the most requested feature in the ASP.NET MVC3 Uservoice site.

If you do decide to start from an empty project template you’ll have to setup everything yourself. That’s not necessarily a bad thing. It provides you the opportunity to setup the project in whichever way you prefer and you can for example use an IoC container to setup ASP.NET Identity which the focus of this blog post.

The main reason why I decided to write this down was because I’ve heard the way ASP.NET Identity is setup being described as “poor man’s dependency injection”, namely the use of .CreatePerOwinContext<T> that you’ll find on the one of the partial class files for Startup.cs in the default template for MVC with Individual User Accounts.

The video where I heard this was ASP.NET Identity Security. The reason why it’s described as “poor man’s DI” is because, instead of actual dependency injection, the dependencies are created in the Startup class, stored as Owin properties (that’s what CreatePerOwinContext does) and then retrieved in the controller (e.g. HttpContext.GetOwinContext().GetUserManager<ApplicationUserManager>()).

Anyway, there’s no reason to settle for a “poor man’s DI”, so lets use StructureMap instead.

Starting from scratch

If you start with the Empty project template File->New->Project->Web->Asp.Net Web Application and then:

Because ASP.NET Identity has dependencies on Owin, and because you probably want to use the owin cookie middleware to handle signing in users, start by installing the Microsoft.Owin.Host.SystemWeb NuGet package. You also need to install the ASP.NET Identity NuGet packages. Here’s the list that you can just copy&paste to the package manager console.

Here we’ll be using the EntityFramework version of the user store for ASP.NET Identity (user store italicized because the interface that exposes the functionality to actually save and retrieve users in ASP.NET Identity is named IUserStore).

install-package Microsoft.Owin.Host.SystemWeb
install-package Microsoft.AspNet.Identity.Core
install-package Microsoft.AspNet.Identity.Owin
install-package Microsoft.AspNet.Identity.EntityFramework

And of course, install the StructureMap package for MVC:

install-package StructureMap.MVC5

Create your Owin Startup class. Easiest way is to just right click on your project->Add New Item->Owin Startup Class.

You should add your own connection string in web.config for the database that ASP.NET Identity will be using. For example, if you want to use LocalDb and have the database called IdentitySetupWithStructureMap your connection string could look like this:

<connectionStrings>
  <add name="IdentitySetupWithStructureMap" providerName="System.Data.SqlClient" connectionString="Data Source=(localdb)\ProjectsV12;Integrated Security=SSPI;MultipleActiveResultSets=True;Initial Catalog=IdentitySetupWithStructureMap;AttachDbFileName=|DataDirectory|\IdentitySetupWithStructureMap.mdf" />
</connectionStrings>

This might not be the right connection string for you, for example your LocalDb might have a different name (you can check yours in Visual Studio’s SQL Server Object Explorer). If you are unsure check this great resource to help you setup your connection string.

StructureMap configuration

Before we dive in the StructureMap configuration for ASP.NET identity it helps to know how you would do it manually, i.e., actually new up everything you need.

The main class in ASP.NET Identity, and probably the only one you need to use is UserManager<T>, where T will be a subclass of IdentityUser (if you derive from IdentityUser you can add your own properties to the user that gets saved, if you did not know about this you should check The good, the bad and the ugly of ASP.NET Identity.

UserManager<T> has a dependency on IUserStore<T>. The implementation we are going to use of IUserStore<T>, UserStore<T>, has a dependency on DbContext (this is the EntityFramework implementation we’ve added through NuGet: Microsoft.AspNet.Identity.EntityFramework).

So “manually” you could do this:

var dbContext = new IdentityDbContext("IdentitySetupWithStructureMap");
var userStore = new UserStore<IdentityUser>(dbContext);
var userManager = new UserManager<IdentityUser>(userStore);

And then you could setup your userManager, for example configure the password validator to only allow usernames with more than 6 characters:

userManager.PasswordValidator = new PasswordValidator
{
    RequiredLength = 6
};

Now that you know how to do it completely manually, if you can pause for a brief moment, go compare this with what gets generated with the default project template when you choose individual user accounts, let me know what you think in the comments.

There’s only four simple things that you need to know to understand the StructureMap configuration for this, they are:

Given a constructor that expects a certain class as a parameter, specify which subclass to use
How do you set properties using StructureMap (for the UserValidator, PasswordValidator, etc in UserManager)
How do select a constructor that is not the default constructor that StructureMap would normally use (one that is not the most specific, i.e., not one with the most dependencies).
How to set the value of value types in the constructor, for example, the value of a string parameter (e.g. connectionString).

Lets start with the setup for the UserManager<IdentityUser>‘s dependency, IUserStory<IdentityUser> (this code belongs inside the StructureMap Registry you are using to setup MVC, usually it’s called DefaultRegistry):

For<IUserStore<IdentityUser>>()
    .Use<UserStore<IdentityUser>>()
    .Ctor<DbContext>()
    .Is<IdentityDbContext>(cfg =>
        cfg.SelectConstructor(() =>
            new IdentityDbContext("value here does not matter, it's only for selecting the right ctor"))
            .Ctor<string>()
            .Is("IdentitySetupWithStructureMap"));

The For Use part is just how you map an interface to an implementation in StructureMap. Then .Ctor<DbContext>().Is<IdentityDbContext>(...) is basically saying, for the DbContext dependency use its subclass IdentityDbContext. However IdentityDbContext has many constructors, and the one we want to use is not the one with most dependencies (which would be the one StructureMap would use).

We want to use the IdentityDbContext constructor that expects a single string parameter, the connection string. We use .SelectConstructor(() => new IdentityDbContext("value does not matter, this is just to let the compiler select the ctor") to do that, and then we use .Ctor<string> to specify the value we want for the string parameter, in this case the value of the connection string: .Is("IdentitySetupWithStructureMap") (if the constructor had several string parameters you could pass the name of the parameter to .Ctor, e.g. .Ctor<string>("nameOrConnectionString") to disambiguate).

That takes care of UserManager<IdentityUser>‘s dependencies. We still need, however, to set the PasswordValidator and UserValidator.

PasswordValidator is the class that defines the minimum password requirements, for example that the minimum password length is 6 characters. UserValidator is the class that checks if the username is valid, for example if you require that the username is an unique email address. There are more properties that you can setup on UserManager, however the way they are setup is very similar to these two.

Because UserManager does not implement an interface we have to use the class directly in our controllers. The way we configure a concrete class in StructureMap is by using the ForConcreteType<T> method, in our case:

        ForConcreteType<UserManager<IdentityUser>>()
            .Configure
            .SetProperty(userManager => userManager.PasswordValidator = new PasswordValidator
            {
                RequiredLength = 6
            })
            .SetProperty(userManager => userManager.UserValidator = 
                new UserValidator<IdentityUser>(userManager));

Here’s the full listing for the Registry:

    public DefaultRegistry() {
        Scan(
            scan => {
                scan.TheCallingAssembly();
                scan.WithDefaultConventions();
                scan.With(new ControllerConvention());
            });


        For<IUserStore<IdentityUser>>()
            .Use<UserStore<IdentityUser>>()
            .Ctor<DbContext>()
            .Is<IdentityDbContext>(cfg => cfg.SelectConstructor(() => new IdentityDbContext("connection string")).Ctor<string>().Is("IdentitySetupWithStructureMap"));

        ForConcreteType<UserManager<IdentityUser>>()
            .Configure
            .SetProperty(userManager => userManager.PasswordValidator = new PasswordValidator
            {
                RequiredLength = 6
            })
            .SetProperty(userManager => userManager.UserValidator = new UserValidator<IdentityUser>(userManager));                
    }

And there you go, with this configuration you can now add UserManager to your controllers as a dependency, for example:

public class HomeController : Controller
{
    private readonly UserManager<IdentityUser> _userManager;
    public HomeController(UserManager<IdentityUser> userManager)
    {
        _userManager = userManager;
    } 
...

I’ve added an example project in Github that uses this setup and a few extra bits. It allows you to create, list and delete users, check it out by cloning the repository

git clone https://github.com/ruidfigueiredo/MvcAndAspNetIdentitySetupWithStructureMapExample

Drop me a line in the comments if you are interested in a summary (cheat sheet) of how to do the most common operations with ASP.NET Identity, for example, create users, delete users, reset a user’s password without requiring email messages, setting up email messages for email confirmation and password reset, etc.

WCF Security Survival Guide – Part 2 – Authorization

March 2, 2016 by Rui Figueiredo 1 Comment

In the first part of this series I described how you can secure your WCF service. The first part describes some of the most common security modes, how you can configure them and how you can add custom authentication.

Part 1 ends using the TransportWithMessageCredential security mode with a custom username/password validator. However, authorization in that scenario is not covered. That’s going to be the focus of this post.

Authorization in WCF

WCF allows us to restrict the access of operations using an attribute named PrincipalPermission, much like what you can do with ASP.NET MVC’s Authorize attribute. For example, this will restrict an operation to users with the Admin role:

[PrincipalPermission(SecurityAction.Demand, Role="Admin")]
public void Remove(int userId)
{
  //...
}

Or if you want to specify alternative roles, for example Admin or Manager:

[PrincipalPermission(SecurityAction.Demand, Role="Admin")]
[PrincipalPermission(SecurityAction.Demand, Role="Manager")]
public void Remove(int userId)
{
  //...
}

If you want to enable more complicated scenarios, for example if you need to check if the user has two roles you need to test that programmatically. I’ll show you that in a second, but first I need to mention the ServiceSecurityContext class.

If you read the first post, and ran the gitub project with the examples you probably noticed that the test service is getting the identity of the user from two distinct sources:

ServiceSecurityContext.Current.PrimaryIdentity
Thread.CurrentPrincipal.Identity.Name

You probably also noticed that when the TransportWithMessageCredential security mode was used we were able to get the current user’s username from the ServiceSecurityContext but not from Thread.CurrentPrincipal. This is because we need to setup a custom authorization policy (because we are using custom authentication).

The ServiceSecurityContext contains the “identity” of the user that is logged in. I mentioned “identity” in quotes because these “identities” are instances of IIdentity. However, the fact that there’s an identity in ServiceSecurityContext is not enough to enable you to use the PrincipalPermission attribute, for example this won’t work even if your username is johndoe:

[PrincipalPermission(SecurityAction.Demand, Name="johndoe")]
public void Remove(int userId)
{
  //...
}

You need to create an custom authorization policy and in it, set the Principal (this will become clear with an example).

Creating a custom IAuthorizationPolicy

The IAuthorizationPolicy interface is defined in the System.IdentityModel assembly (it’s the same that contains UserNamePasswordValidator) and is:

public interface IAuthorizationPolicy : IAuthorizationComponent
{    
  ClaimSet Issuer { get; }
  bool Evaluate(EvaluationContext evaluationContext, ref object state);
}

And IAuthorizationComponent just defines an string getter for a property named Id.

The Id needs to be unique, but can be whatever you want; I couldn’t find information about ClaimSet, all the examples I’ve found just implement it as return ClaimSet.System;, so if you have more information about this, please let me know in the comments, otherwise… just return ClaimSet.System.

The evaluate method, if it returns true it stops other authorization polices from being evaluated (there can be multiple authorization policies, that will become clear when we create the configuration to add our own configuration policy).

The three important questions that need to be answered regarding what you need to do in your implementation of this interface to enable authorization are:

Where is the custom authenticated user’s username?
What do I have to do in the IAuthorizationPolicy‘s Evaluate method to enable the use of the PrincipalPermission attribute in the service implementation?
What do I have to do in the IAuthorizationPolicy‘s Evaluate method to enable the use of Thread.CurrentPrincipal

1. Where is custom authenticated user’s username?

In the EvaluationContext instance that is passed in as a parameter to the Evaluate method there’s a property named Properties that is of type IDictionary<string, object>.

In that dictionary you’ll find a key named “Identities” which contains a List<IIdentity> instance. Usually it will only contain one element. If you are using a custom UserNamePasswordValidator (see Part 1) and the the class name is MyCustomUserNamePasswordValidator you’ll find one item in that list whose authentication type is “MyCustomUserNamePasswordValidator”.

Here’s how you can get to the instance of that IIdentity that represents the authenticated user, using MyCustomUserNamePasswordValidator:

var identity = (evaluationContext.Properties["Identities"] as List<IIdentity>).Single(i => i.AuthenticationType == "MyCustomUserNamePasswordValidator");

The username is here: identity.Name.

2. What do I have to do in the IAuthorizationPolicy’s Evaluate method to enable the use of the PrincipalPermission attribute in the service implementation?

You have to set a property named “Principal” in evaluationContext.Properties to an instance of an IPrincipal. The particular type of IPrincipal implementation you choose is not important, because all of them will implement IsInRole, and that’s what is important to enable the use of the PrincipalPermission attribute.

Here’s an example of creating a ClaimsPrincipal with the identity fetched in (1) and adding a role named Admin:

var claimsIdentity = new ClaimsIdentity(identity);
claimsIdentity.AddClaim(new System.Security.Claims.Claim(System.Security.Claims.ClaimTypes.Role, "Admin"));
var claimsPrincipal = new ClaimsPrincipal(claimsIdentity);

evaluationContext.Properties["Principal"] = claimsPrincipal;

3. What do I have to do in the IAuthorizationPolicy’s Evaluate method to enable the use of Thread.CurrentPrincipal

Just set it, for example:

Thread.CurrentPrincipal = claimsPrincipal;

And it will be available in your service’s operations. The reason for (3) is that if you want to enable scenarios where you want to check if your user has both the role “Admin” and “Manager” you can do this:

[PrincipalPermission(SecurityAction.Demand, Authenticated = true )]
public void Remove(int userId)
{
  var principal = Thread.CurrentPrincipal;
  if (!(principal.IsInRole("Admin") && principal.IsInRole("Manager")))
    throw new System.ServiceModel.Security.SecurityAccessDeniedException("Insuficient privileges");
  //...
}

A custom authorization policy complete example

public class CustomAuthorizationPolicy : IAuthorizationPolicy
{
    public bool Evaluate(EvaluationContext evaluationContext, ref object state)
    {
        var identity = (evaluationContext.Properties["Identities"] as List<IIdentity>).Single(i => i.AuthenticationType == "CustomUserNamePasswordValidator");

        var claimsIdentity = new ClaimsIdentity(identity);            
        claimsIdentity.AddClaim(new System.Security.Claims.Claim(System.Security.Claims.ClaimTypes.Role, "Admin"));
        var claimsPrincipal = new ClaimsPrincipal(claimsIdentity);
        evaluationContext.Properties["Principal"] = claimsPrincipal; 

        Thread.CurrentPrincipal = claimsPrincipal;
        return true;
    }

    public System.IdentityModel.Claims.ClaimSet Issuer
    {
        get { return ClaimSet.System; }
    }

    public string Id
    {
        get { return Guid.NewGuid().ToString(); }
    }
}

Configuration for the new Authorization Policy

To add the new authorization policy we need to edit our WCF configuration and add it as part of a serviceBehavior, namely through serviceAuthorization, whose principalPermissionMode we must set to Custom, here’s an example for the configuration:

<behaviors>
  <serviceBehaviors>      
    <behavior>      
      ...   
      <serviceAuthorization principalPermissionMode="Custom">
        <authorizationPolicies>
          <add policyType="TheNamespace.CustomAuthorizationPolicy, TheAssemblyName"/>
        </authorizationPolicies>
      </serviceAuthorization>
      ...          
    </behavior>
  </serviceBehaviors>
</behaviors>

You can find a working example here: https://github.com/ruidfigueiredo/wcfsecuritysurvivalguide-part2-authorization

If you would want to see a more elaborate custom UserNamePasswordValidator and IAuthorizationPolicy, maybe using ASP.NET Identity, please let me know in the comments so I can decide if it’s worth dedicating some time to write about that.

Skip links

Rui Figueiredo

Practical tips for creating your own questions/answers

Conclusion

The “unit”

A little taste of test first development

The dependencies

Tip 1. Starting point: The Url

Tip 2: Chrome developer tool’s inspector (Elements tab) for HTML/CSS

Tip 3: Markup injected in the page via Ajax

Tip 4: Plain JavaScript events

Tip 5: jQuery events

Tip 6: Client side frameworks (KnockoutJS and AngularJS)

Tip 7: Leverage the extensibility points of the Framework that you are using to inject debug information when rendering the pages

Owin Middleware

Owin/Katana and ASP.NET Core, MVC6 and MVC5

ASP.NET MVC Client Side validation requirements

Getting away with using Client Side validation without a model

Example

1. Sharing layout pages and styles

2. Links between pages in different sub web applications

3. How to setup your development environment (machine key, local IIS)

4. Setup authentication and authorization in each of the web applications

5. Share components between different sub web applications

Conclusion

Starting from scratch

StructureMap configuration

Authorization in WCF

Creating a custom IAuthorizationPolicy

1. Where is custom authenticated user’s username?

2. What do I have to do in the IAuthorizationPolicy’s Evaluate method to enable the use of the PrincipalPermission attribute in the service implementation?

3. What do I have to do in the IAuthorizationPolicy’s Evaluate method to enable the use of Thread.CurrentPrincipal

A custom authorization policy complete example

Configuration for the new Authorization Policy