Alexander Beletsky's Development Blog: 2011-05

Inside ASP.NET MVC: Setting up project for hacking

As we already got debug assembly and pdb file, it it time to create a test project that would be a sandbox for all experiments. This would be classic MVC application, but we going to substitute release version of MVC framework with our locally built version, which allows us easily debug it.

Create new project

Let's create new project. It would be MVC 3 Application:

new project

On a second step just select "Empty" in "Select template" dialog. "Razor" would be for view engine.

Correct system.web assemblies section

Go to web.config of a new project and comment out System.Web.Mvc and System.Web.WebPages assemblies. By doing that we saying: "those 2 assemblies are no longer a part of compilation of our application".

web.config

UPDATE: If you do just that, later you will get an exception from Razor assemblies, about wrong type cast. To avoid that, it is even better to remove all assemblies in this section, plus Find/Replace PublicKeyToken=31bf3856ad364e35 to PublicKeyToken=null. That would do not make application to search GAC for missing assemblies and load locally built ones.

Correct application references

Now, remove existing references for System.Web.Mvc and System.Web.WebPages and put locally built ones instead. Right click on references project section and browse to the Debug folder of mvc3-rtm build directory. Try to build application now, it should be build without any issues.

setup

Home controller

Further steps are pretty simple, just add a test controller - HomeController with Index action. Add just a simple view for that.

controller

Check it out!

Place a breakpoint into action method and press F5 to start debugger. Now you we've got a full stack trace that allows to see every aspect of MVC world.

stack

Conclusion

The setup is easy and allows to jump in very quick. If you got any issues with that, please let me know.

What is next?

Next I will hack the entry point of MVC application called MvcHandler and understand how it works.

Please stay tuned.

Inside ASP.NET MVC: Packing bags for journey

I'm big fan of ASP.NET MVC framework. It is smart, clean and testable. A lot of stuff you can do with MVC are very intuitive, you don't have to go thought pages of documentation to do some basic stuff. I do not use MVC on my primary work yet (unfortunately) but my small project became very nice sandbox to play around and learn stuff.

I definitely learned something thought the year of hooked up with MVC. But feel I still miss a lot of details. Moreover, my hackers curiosity always force to look - "what is inside?".

Great stuff about ASP.NET MVC project is that is open source. It is been released as open source and remain open source project till version 3.0.. In fact, it is very easy to start learning framework from inside! This what I'm going to do. I would like to go depth and hope that journey would be interesting and valuable.

Today I'll to prepare everything I need for journey. I would be happy if you join me, so we would help each other along the way :).

Getting out source code

That's easy. Just google for "MVC3 source code" and it would bring you directly to ASP.NET Codeplex site.

Download mvc3-rtm-sources.zip. Unpack the archive and you will see the solution file WebRuntime.sln and 2 folders: mvc3 and webpages. Layout of both folders are similar:

    /--- bin
    /--- src
    /--- tests
    /--- tools

Source code package is pretty much organized. Solution does not using NuGet, btw :)

Meet the solution

It's time open up solution! Fire up Visual Studio and open WebRuntime.sln. Let the journey begins!

Nothing fancy there. Minimal number of solution folders, just MVC and WebPages. Each of folder contains a bunch of projects in it. In nearest future we would be much interested in one System.Web.Mvc - heart of MVC framework.

solution

Build it

Don't waste any time and just build the solution. As it usually happened to me, with OSS projects - you always have some issues during build. It is not the case for MVC. Build runs smoothly as possible. The output is being placed in mvc3/bin/debug folder. So, as soon as build finished check it out and see for content.

output

Run tests

All TDD lovers would be really pleased with a number of tests in solution. There is 5004 tests actually, solid number of tests. No surprise MsTest framework is used. Let's just try to run them!

In menu Test -> Run -> Tests in Current Context. Here we go.. It took a bit time to run all. I was a little disappointed to see that some tests are failing.

I did an inspection of failures and found out that all of them are actually related to my Local Settings. I use "Ukrainian Locale" that means time format, decimal point symbol etc. are differs from "United States Local". It's a quite common issue of units, workaround could be a simply change your locale to the one that rest development team uses. The rests of tests run's just fine!

tests

Conclusions

So, my initial impression are:

  • Easy to build and start up
  • Code looks really nice
  • Code style a little differs from default C# especially in with "{"
  • code
  • Tests following AAA pattern really nicely
  • test
  • Code contracts are not used, but code pretty defensive with check for null arguments etc.
  • exception
  • All members are with "_" underscore - I like that :)
  • Methods are small and clean

What I don't like to far:

  • Solution is quite big - I would be happy to have separation between MVC part and WebPages (if it is possible)
  • Tests are really slow. Suite run for 3-4 minutes (upd: I blame MsTest runner for that, with TD.NET all tests passed for 89 sec's)
  • Already found some comments that lie (one more point to try to avoid comments if possible)

What is next?

Next, I will setup a test project which allow us to fully get in deep of details and see how it work.

Please stay tuned.

.NET Saturday in Ciklum

This Saturday it was really nice event organized and hosted by Ciklum. I had a big pleasure to be part of the event and was given a speech about Continues Production (Delivery) and my experience of using UppercuT, RoundhouE and Jenkinks for that.

The organization of event was really nice. Only the speakers were one who introduced disorder there :). Some speeches were bit longer than planned, that's why schedule ruined from the very beginning of event. My speech was planned for 0.5 hour, but I've been 1 hour on stage.

I've been pretty nervous since I never gave public speeches, but I really liked that experience. I hope to visit/participate more .NET events in future. At the end I was happy that everything is alright and I'm finally done. So, in great mood and relaxed I stayed to listen for the rest of speakers.

Also I was really nice to meet Sergey Kalinets, Anton Vidishchev who shared their experience in TDD and ASP.net MVC framework. Such type of event are really great for professionals gathering, helping to better communication and knowledge sharing.

Let me wish Ciklum and Inna Tueva in particular to have will and energy for organizing such events. Thanks a lot!

Continuous Delivery: Make it work

As soon as we setup project infrastructure in a way that build and deploy is done by one batch command, we are ready to run this process continuously. It is possible to take any CI server you want, I stopped on Jenkins.. basically it contains everything I need. But before start with Jenkins, let's one more time take a look current infrastructure and goals.

Git branching model

There is great article on that topic, I used main ideas from that. So, I have 2 branches in my origin that exists all the time. They are: master and development. All pushes I do during implementation are going to development branch. No direct pushes are made to master at all (there are some exceptions of cause, but I try to follow that rule). As soon as code in development is stabilized, I prepare special release branch (update version and small clean up there) and release is being merged to master as it tested and everything is fine.

Staging and Production environments

I define 2 types of environments: Staging and Production. Environment includes: binaries, markup, database, deploy scripts.

Everything that is being developed are goes to staging environment immediately (and automatically). Production update is being run manually as soon as all testing on Staging are finished. Separation of environments is great idea. Even if we are trying hard to have potentially shippable software during each build, this is not true. It is usually a lot of problems in latest version that need to be fixed before production server update.

So, Staging is a result of build/deploy of development; Production is a result of build/deploy of master.

In my case, everything you see here is a result of build of this. Everything you see here is a result of this.

What I want to get?

I want to have automatic system that would update my Staging with every push to development branch and I want automatic system that would update my Production as soon as master branch is ready to. As it said in overview having such system is great reduce of deployment overhead. Let's make this happen!

As I said earlier I would use Jenkins. Great stuff about this software that is very intuitive! Setup is easy and fun, I would put just a little guidance of process.

Local Jenkins server setup

As you download Jenkins WAR file, it is easy to start just with java -jar jenkins.war. Now, the dashboard is available at http://localhost:8080/.

Plugins

First, you should add all required plugins. It already contains some, but some are missing. I've add such to my default configuration:

Go to "Manage Jenkins" -> "Manage Plugins" and install missing one.

So, my configuration looks like this:

plugins

Configure system

Now go to "Manage Jenkins" -> "Configure System" and find "Git" section there. You should provide with your user name and path to Git. I have git in my PATH variable (recommend you do the same).

Create new job

After basic configuration is done, let's define new job. Go to "New job". In common section you should give your job name and provide with githib path to repository.

common section

In Source Control Management: URL to Git repository, Branch to build and Repository browser (optional).

SCM

Build triggers (explain a bit later).

build-triggers

In Build section you specify commands to build/test/package and to deploy results. See my previous post for details.

build

Finally in Post-build section, specify path to grab artifacts.

Test it locally

As soon as configuration is done, you can test your build locally. Just start your job and make sure that: build run, tests executed, new database deployed, new application deployed. Basically if everything done correctly in previous steps you should not experience any troubles. If it is fine that means the configuration is simply ready to be put to production server.

Move it to production

As it defined, our goal is to update staging and production with out any manual work. The simplest scenario then CI server resides on the same machine as production server.

Great thing about Jenkins is that it requires no installation. So, I packed all content of ~/User/Jenkins folder and Git into one single package and FTP'ed to my production server. As you unpack, configure Git in PATH variable, setup up your private repository keys it is just simply ready to work. Now, test it remotely - staging and production should be able to update with one single click on Jenkins dashboard.

Little adjustments for production server

My VPS has Windows auto-update enabled, so It is being rebooted from time to time. As it is being rebooted, you should start Jenkins manually. Fortunately it has really nice feature - Jenkins might be run as Windows Service. This is extremely useful, I recommend to do it from very beginning.

Using Github hook to automate build run

I want my every commit to development to be tested and to be visible on Stage. As it said in Configure system part, you can setup build triggers. With build trigger you can run build just by requesting this URL http://jenkins.server/jobname/build?token=TOKEN.

In github repository go to Admin section and setup new hook in "Post Recieve URLs" section, like this:

github

This concludes the setup.

Control your builds all the time

What we did is only very basic setup, but it already gives a lot of power. It is really easy to track all builds/artifacts/history of changes. Downtime of site is extreamly slow (comparing to manual changes). Dashboard is very convenient to control builds:

dashboard

Continuous Delivery: Setup and run

In my previous post I tried to make it clear that continuous production is for good. This post I'll show how to prepare environment for continuous production. The idea is that you should be able to configure and test it locally. All configurations have to be part of source code under SCM. It should not depend on machine and run any environment you like. Success criteria is: pick up clean machine, do checkout, run build.bat/deploy.bat and have installed web application.

Integration and database deployment

As I said, UppercuT and RoundhousE are really nice tools for that. As soon as you follow the instruction's you will have a build.bat, that would be able to build up all binaries, run tests against that and put all build artifacts to package. That is pretty good for start, but we still missing "deployment" part.

As you a little bit more familiar with UppercuT, it provides good facilities for deployment as well. Basically, there are folder deployment\templates\ where you able to define your custom deployment scripts. Typical web application requires 2 scripts:

  • AppDeployment.bat - for web site deployment
  • DbDeployment.bat - for database deployment

There files are templates, from which script for particular environment is generated. The environment is defined in settings folder and include such information as deploy folder, web site name, database name, server name as number of variables. Example,

<?xml version="1.0" encoding="utf-8" ?>
<project name="Settings">
  <!-- environment settings -->
  <property name="environment" value="PRODUCTION" />
  <!-- servers -->
  <property name="server.database" value=".\SQLEXPRESS" />
  <property name="web.deploy.folder" value="c:\trackyt.net\web\" />
  <property name="web.site.name" value="trackyt.net" />

  <property name="database.name" value="trackytdb" />
  <property name="log.level" value="DEBUG" />
  <property name="app.user.name" value="alexander.beletsky" />

  <!-- base settings -->
  <property name="project.name" value="trackyt.net" overwrite="false" />
  <property name="repository.path" value="git://github.com/alexanderbeletsky/trackyt.net" />
  <property name="folder.app.drop" value="${project.name}" overwrite="false" />
  <property name="folder.database" value="db" overwrite="false" />

  <!-- database deployment -->
  <property name="dirs.db" value="..\${folder.database}" />
  <property name="file.version" value="_BuildInfo.xml" overwrite="false" />
  <property name="restore.from.path" value="..\${database.name}.bak" overwrite="false" />
  
</project>

In template .bat file it is possible to refer, to some particular variable, so it is possible to make those quite generic. After build, template .bat files are post-processed and actual batch is generated. The name would be like ENVIRONMENT.AppDeployment.bat, where ENVIRONMENT is type of environment you defined.

Web site deployment script

If you do ASP.net (MVC) website in 99.9% cases you will be happy with simple XCOPY deployment type. Basically it means, simple copy of website to defined IIS folder.

But, as soon it is continuous production deploy it means that Web Site is already running. It would not be possible to re-write some files, since they could be used by IIS. So, we need to stop the site before update. I found very good possibility for that with %windir%\system32\inetsrv\appcmd command. After site is stopped, we just copy full content of Web folder, remove some redundant files and run site again. In batch code it would look like,

@echo off

SET DIR=%~d0%~p0%

SET web.deploy.folder="${web.deploy.folder}"

echo stopping web site..
call %windir%\system32\inetsrv\appcmd stop site ${web.site.name}
if %ERRORLEVEL% NEQ 0 goto errors

echo copy application content
rmdir /s /q %web.deploy.folder%
xcopy /E /F /H /R ..\_PublishedWebSites\Web %web.deploy.folder%
xcopy ..\build_artifacts\_BuildInfo.xml %web.deploy.folder%
if %ERRORLEVEL% NEQ 0 goto errors

echo remove redudant files
del %web.deploy.folder%*Tests*.htm*
del %web.deploy.folder%Web.Debug.config  
del %web.deploy.folder%Web.Release.config 
del %web.deploy.folder%*packages* 
if %ERRORLEVEL% NEQ 0 goto errors

echo starting web site
%windir%\system32\inetsrv\appcmd start site ${web.site.name}
if %ERRORLEVEL% NEQ 0 goto errors

goto finish

:errors
EXIT /B %ERRORLEVEL%

:finish

Database deployment script

RoundhousE does all infrastructure work for us. All we need to create a batch file, that would be able to run during continuous production cycle. As well as AppDeployment.bat I defined DbDeployment.bat in deployment\templates\ folder. But before any update of database it is always good to have a backup, to be able to restore from it if something went wrong. Actually, RoundhousE should have such ability, but unfortunately I didn't get how to use it. I've created my simple SQL script that is able to backup.

USE $(Database);
GO
BACKUP DATABASE $(Database)
TO DISK = 'C:\backup\$(Database).bak'
   WITH FORMAT,
      MEDIANAME = 'C_SQLServerBackups',
      NAME = 'Full Backup of $(Database)';
GO

And corresponding batch file, that would run backup.sql.

@echo off

if '%1' == '' goto usage
if '%2' == '' goto usage

sqlcmd -S %1 -i .\scripts\backupdb.sql -v Database = %2 -e
if %ERRORLEVEL% NEQ 0 goto errors

goto finish

:usage
echo.
echo Usage: backup.bat [server] [database]
echo [server] - server eg. mymachine\SQLEXPRESS
echo [database] - name of database to backup
echo.
EXIT /B 1

:errors
EXIT /B %ERRORLEVEL%

:finish

Both files are placed into deployment\scripts folder. So, the DbDeployment.bat template, would first run database backup and if it is successfull, run RoundhousE to update database.

@echo off

SET database.name="${database.name}"
SET sql.files.directory="${dirs.db}"
SET server.database="${server.database}"
SET repository.path="${repository.path}"
SET version.file="${file.version}"
SET version.xpath="//buildInfo/version"
SET environment="${environment}"

echo backup database
call .\scripts\backupdb.bat %server.database% %database.name%
if %ERRORLEVEL% NEQ 0 goto errors

echo update database
"%DIR%rh\rh.exe" /d=%database.name% /f=%sql.files.directory% /s=%server.database% /vf=%version.file% /vx=%version.xpath% /r=%repository.path% /env=%environment% --ni --simple
if %ERRORLEVEL% NEQ 0 goto errors

goto finish

:errors
EXIT /B %ERRORLEVEL%

:finish

Putting it all together

We already have build.bat as part of UppercuT, now we need to define deploy.bat that would do deployment of product. It would be called immediately after build.bat finished, so binaries are ready, tests passed and code_drop folder contains all artifacts for deployment. The script is rather simple and utilize stuff we did previously.

@echo off

if '%1' == '' goto usage

SET ENV=%1

cd .\code_drop\deployment

echo Deploy database
call .\%ENV%.DbDeployment.bat
if %ERRORLEVEL% NEQ 0 goto errors

echo Deploy application
call .\%ENV%.AppDeployment.bat
if %ERRORLEVEL% NEQ 0 goto errors

goto finish

:usage
echo.
echo tracky.net deploy script
echo Usage: deploy.bat [environment]
echo [environment] - deployment environment could be STAGING or PRODUCTION
echo.
EXIT /B 1

:errors
echo Build FAILED
EXIT /B %ERRORLEVEL%

:finish
echo Build SUCCESS

Notice, it receives the parameter ENV. It will contain type of environment for deployment. For staging environment, you should call deploy.bar STAGING, for production deploy.bat PRODUCTION.

Testing it out

That's basically it. Now, you should make sure everything works as expected. Run build.bat/deploy.bat, make sure build went with no errors, deploy.bat correctly does back up of database, updates database and update site content.

As I said on top, it is very important that configuration is part of product, part of source code. If you follow this, it will be possible to deploy application but just getting sources from SCM. This is first step of setting up your Continuous Production server.