Continuous Integration with TeamCity – Supporting Multiple Branches

Everybody will agree that nowadays Continuous Integration (CI) process established on project is as usual as Version Control System (VCS) for source code. Actually both are tightly coupled – CI server takes source code from VCS and runs all the pre-configured by build engineer magic: source code compilation, automated tests run, installation package preparation, etc.

Here at ELEKS we get used to TeamCity. In this and following posts I’ll share our experience with using TeamCity. When mentioning VCS I’ll refer to Git. Still, most of described also apply to Subversion and other VCS.
First common problem that we’ll discuss – establishing CI process for multiple branches of source code: usually for development and release branches.

The beginning

Development is started. Source code repository setup completed. After several days of development, when there is something to build, we add a TeamCity Project and several configurations to this Project:

TeamCity

Time flows; more code is being committed into source control. More TeamCity configurations are being added:

TeamCity

After half a year of development we already have 40 configurations. These configurations were created and evolved together with source code. Because all the time we worked only on single development branch (called master) everything was simple and fine. But now we have to make code-freeze and introduce a release branch which brings new problems to solve:

  1. We have to run continuous integration process for both master and release branches.
  2. From time to time we have to merge fixes from release branch into master branch and vise-versa.

Continuous Integration for release branch

For a single branch we had 40 TeamCity configurations. For two branches we must have 40 + 40 = 80 ones. Or can we have only 40 configurations but run them either for master or release branch? Short answer: we didn’t find a reliable way how to do this. Besides, it is nice to have configurations for different branches running in parallel and for this feature we must have a copy of these configurations so each branch can be independent.
How can we quickly get these 40 extra configurations? At the time we make a release branch from master branch it is obvious that they must be the exact copy of existing 40 configurations developed for master branch. Thankfully, TeamCity offers a way to clone whole Project with all its configurations.
Before cloning Project1 project make sure that VCS root referenced by configurations of this project is shared. Look at the bottom of Edit VCS Root page:
TeamCity

Now go to Project1 settings:

TeamCity

Click Copy:

TeamCity

Name new project as Project1-release and confirm. Rename old Project1 to Project1-master.

TeamCity

At this point Project1-release is the exact copy of Project1-master. The last thing to do is to change VCS root branch. Click on any configuration in Project1-release (for example on build-app-debug), click Edit Configuration Settings and select Version Control Settings:

TeamCity

Edit VCS root – change VCS root name to repo-release and Ref name to release (Ref name stands for branch name there). Scroll to the very bottom of Edit VCS Root page:

TeamCity

Make sure “Apply to all templates & configurations of Project1-release project where this VCS root is used (a copy of this VCS root will be created)” option is selected! Click Save.
Now all configurations from Project1-release project have VCS root that points to release branch. If you have more than one source code repository, for example you are using git sub-modules, you have to perform VCS root editing steps for each VCS root.
Please note that described technique works as long as all your configurations reside within single TeamCity Project! If, for some reason, you split configurations between two TeamCity Projects (say Project1-build and Project1-tests) – you are in trouble when applying copy project technique, especially if you have more than one VCS root.
Congratulations. We have two TeamCity projects and two VCS roots, each references different code branch. We can commit code either to master or release branch and run corresponding TeamCity configurations in parallel (…if you have multiple Build Agents, otherwise they’ll be queued).
Of course having two almost exact clones of TeamCity configurations doesn’t feel “right”. But it works!

New development and new release

First version of the product released. Developers continue to work on master branch. They add new code, write new tests. As a result more TeamCity configurations are created; some existing configurations are changed (extra build step added, some steps changed, new artifacts dependencies used, etc.). Master branch and Project1-master TeamCity project are evolving together – when some TeamCity configuration is broken build engineer fixes it. What is important is all this evolving process – source code is under source control but your TeamCity projects and configurations are not.

Sometimes we make fixes to release branch. Usually these are only bug-fixes, no new functionality is introduced – no reason to touch Project1-release configuration.

Three months later a new release comes. Team lead merges code from master branch into release branch and… a lot of configurations in Project1-release project do not work! Moreover, some configurations are missing. Of course they are missing – they were added to Project1-master but not to Project1-release.

Here is the most simple and reliable way of fixing Project1-release – just remove it and create a new copy of Project1-master! But consider following side effects: Project ID, Configurations IDs and VCS root IDs will change. If your automation tools rely on these IDs you have to reconfigure these tools. Actually, these are ways to preserve VCS root IDs – the most quick and reliable solution envisions manual editing of TeamCity project XML configuration file and TeamCity server restart.

Alternatively you may reapply changes made in Project1-master to Project1-release manually using TeamCity WEB interface. But if number of changes is high the work quickly becomes tedious and error prone.

Conclusion

In this post we reviewed one of the possible ways of supporting CI process for multiple branches of source code. I do not say that described approach is ideal and you should immediately start using it; I blogged about it because it worked (and works) well on rather large project. We tried other ways of achieving flawless multiple branches CI – for example using TeamCity templates feature. But time showed that “clone whole project and VCS roots approach” is the most optimal solution by complexity/features criteria. If you’ve successfully applied “better” solution that solves multiple branches support issue – please share it in comments.

13 responses to “How Azure Web Sites Sucked in Production”

  1. sestocker says:

    WordPress on Windows is almost always a bad idea. But to your point, these limitations need to be spelled out ahead of time with user warnings via email. I don’t run any any high traffic (or even moderate traffic) WordPress sites so shared plans work (I like NearlyFreeSpeech). Media Temple has a compelling WordPress plan where core updates are auto applied and there is a staging environment: http://mediatemple.net/webhosting/wordpress/

  2. nonuby says:

    I dont see a problem here. PHP limit is per process/thread (practically same thing in linux), which is generally a thread per request, with a dog like wordpress issuing serial mysql queries per request then and say each request takes 250ms to complete then just 16 req/sec would fcuk. Dont blame Azure for PHP request model sucking.

    • Yuriy Guts says:

      Agree, we should have load-tested better. But what I blame Azure for, though, is not mentioning the query quota at all, as well as not mentioning the fact that the associated ClearDB plan is free. Basically the only thing the customer sees in the documentation is the 20MB storage limit. Getting to the ClearDB portal from the Management Portal is not that obvious, unfortunately.

  3. nonuby says:

    ClearDB freemium plan of course has limited, just like Firebase, noted in the FAQ. Of course if you missed this you’d have picked it up in real world testing (with blazemeter or otherwise). 500 generally indicates a problem with your code, possibly session handling presuming frontend had sticky session which may of not been the case (hence periodic issues?). I think you’re fast to blame Azure here.

  4. damienguard says:

    WordPress is a system hog. Without W3Supercache or TotalCache it won’t handle any kind of moderate load. This is especially more noticeable on Windows as it was never optimized for IIS and barely runs. Perhaps the title should be “How Azure Web Sites Sucked in Production for WordPress” because for .NET applications they’re awesome.

  5. Ken Smith says:

    For what it’s worth, I’ve had pretty good luck with Azure websites, but I’ve always run .NET code on them. Given that probably 99% of all WP websites run on Linux, I would be surprised if running it on Windows didn’t have some significant gotchas. LAMP is supported on Azure, but it doesn’t seem to be quite the first-class citizen that .NET is.

  6. Rob Conery says:

    As Damien mentions above – you can’t use WordPress in any capacity (especially in production!) without rolling in some kind of caching. Cloudflare, TotalCache – *something*. This would have mitigated much of your trouble. Also – yeah a shared plan on a production site with 10K visits using WordPress… I might suggest reading the limits of said plans before you push live. I have been bitten as well, but the shared plan is rather stingy.

    • Yuriy Guts says:

      @Rob: I must admit that the 10K pageviews statement sounded a bit too loud. This was the kind of traffic we got because of a HN effect that day. Usually, when we don’t publish new content, the traffic is much, much lower.

  7. michelebusta says:

    We’ve been very successfully running Azure Websites for large scale customers for over 1 year now. The trick is, you have to get a proper VM, standard instance. As with any cloud deployment, database retry is a necessary evil, and understanding the usage patterns of the site so that you can ensure the appropriate VM size. I agree that the free version isn’t acceptable for most anything if you have traffic, quota shutdown sucks.

    • Yuriy Guts says:

      Yep, we use that sort of environment for our corporate website. Though we launched it when WAWS was still in preview, so it’s just a standard Web Role on a dedicated VM.

  8. ruslany says:

    Yuriy, the recommended way to run a production site on WAWS is to scale it to Basic or Standard mode which would put the site on a separate VM and will remove any quota limits. As with regards to the database outages – I had the same problem when I was migrating my WordPress-based blog onto WAWS. I eventually upgraded the database to a paid option. It looked like the free ClearDB option was only suitable for a development usage, but ran into limitations under more or less real production load.
    Thanks for the feedback on the quotas functionality for shared sites – we will look into ways to improve it.

    • Yuriy Guts says:

      Thanks for the advice Ruslan. The main motivation for using a shared plan was a modest volume of traffic we normally serve, as well as the fact that for two years our blog has been able to work on a shared hosting (I assume Blogger is) without interruptions. Looking back at all this: we made quite a few mistakes on our part, although I wish Azure documentation provided more details about ClearDB limitations (unfortunately, the pricing page doesn’t even mention the existence of ClearDB at all). This might have eliminated a lot of confusion for us as first-time ClearDB users. Anyway, we’ll be very happy to see the shared mode evolve since it’s still an awesome place to start building new projects.

  9. whyJoe says:

    We were just in the same boat. As far as all the IIS/Windows comments, or the you-should-have-read-the-limits comments… 1) I’ve run WordPress on Windows shared hosting on many other low-cost providers without these issues. I always have had Windows hosting plans. 2) Yeah, ok, the limits are spelled out somewhere. But the point is, the limits are too low, making it an uneconomic solution for a small-traffic site. Other shared hosting providers offer a much better value. On Azure, there is basically no way to do WordPress other than a VM where you can install your own MySQL instance (even though that’s less than ideal for a number of reasons.) I hit ClearDB’s low query limit even on a PAID plan, JUST with developer traffic.

Leave a Reply

Your email address will not be published. Required fields are marked *