Dynamics – setting up the dev process, Part 2

Frankly, I got a bit stuck on this. As far as Dynamics solution development process is concerned, I’ve seen quite a few different scenarios. Some of them were more mature than others, but none of them were without problems.

Merge problem aside, you may have multiple Dynamics organizations with one org per developer or you may have just a single development org, you may or may not have a dedicated QA/UAT/Staging, you may or may not have the source control, you may or may not have automated testing, you may be following Waterfall or you may be following Agile..

And the choice is not, always, yours. For example, depending on the licensing situation, and, also, depending on the infrastructure and project limitations, you might or might not be able to spawn multiple Dynamics organizations and/or instances. This is not to mention that clients purchasing Dynamics are not necessarily expecting that there will be any development involved, so they might not be even thinking of provisioning the source control at all.

Maybe there is just no one process that fits all, then, so it probably does not make sense to look for the Holy Grail here, but it may still may make sense to look at what tools and approaches we have at our disposal and how to use them so the whole process works better, not worse (if and when we have those tools available).

For example, I’ve already mentioned “merge” problem before, but, come to think of it, it’s a bit of a made up problem.

Imagine the environment with only one organization, which is your production organization. If every consultant were working in the same organization, there would be no merge problem at all. Sure that would lead to a whole set of other issues related to concurrent work, but there would be no need for merging.

We introduce the merging issue once we choose to have multiple Dynamics organizations in our environment, and we do that because we want to avoid different consultants stepping on each other’s toes. Which seems to be a rather straightforward answer to the problem of concurrent work, but, apparently, it brings another problem.
Point being: depending on what we need more in each particular situation, we may have to resort to different tools and approaches. So what are those, what problems do they solve, and what problems do they introduce?

Here is my uncategorized list of those tools (and related notes) then.

1.    The methodology

Is it going to be Agile, or is it going to be Waterfall?

From the Dynamics perspective, a lot of that is about the amount of customizations and configuration we’ll have to keep in the pre-production environments before we can release them to production. It’s literally impossible to follow the Waterfall when we only have one organization(production), so this assumes we’ll have at least two organizations.  Which is reasonable – even in the online model, we will usually have Sandbox available for this.

But it does not end there. Imagine that we have features which will be delivered at different times. For example, there could be a module that is going to cost us a couple of months of development, but there will be a few smaller modules that we will be developing in parallel. Different consultants will be working on those.

If there is any level of intersection between those modules, we may not be able to deliver those smaller modules to production if we choose Waterfall approach where only a fully-functional product will ever be delivered since we’ll have to wait for two months. Yes, we may do longer-term development in a separate organization, but we’ll end up with that dev organization being completely out of sync from production, so it’s just introducing other problems.

Still, if your users don’t want to deal with half-cooked functionality, maybe Waterfall is the way to go, and, then, you may have to plan your releases accordingly to avoid too much merging.

However, if the organization you are working in is open to a more agile approach, it might consider allowing those features which are still in development to be delivered to production before they are fully ready. That may require some extra work from the team to ensure that the end users don’t get confused, so there may be more emphasis on setting up security roles, delivering demos, and providing training. All of that is going to take time from your dev team and from the end users, though.

And what’s my personal favourite? I think Dynamics is not meant for the Waterfall – everything is just way too fluid and dynamic there, so it may be better to take a more agile approach from the beginning. Mind you, the client will have to make an effort, too. Really it’s all about getting all configurations and customizations at least to the integration environment as soon as possible to make sure no one is getting out of sync too much. Worst case scenario – anything up to and including the staging environment can be part of the agile process. Delivery to production may happen in a more waterfall-ish way, once certain milestones have been reached.

2.    One Dynamics organization vs multiple organizations

The main problem with having one organization only is that everyone working there has to be mindful of the work others are doing. The main problem with having multiple organizations is that there is no easy way to merge configuration changes from different development organizations in Dynamics.

In other words, it’s almost as if there were some amount of pain and all you could do is just distribute the pain between those two problems (concurrent work and merge).

Still, there are some ways to make it easier.

As far as concurrent work is concerned, it’s often possible to assign different pieces of work to different consultants. If there is not a lot of interference, concurrent work might not be an issue at all. If there is some interference, those consultants may just have to talk more often.

On the other hand, if there is a piece of functionality that has to be developed in isolation, the best way to do it might be to create a separate organization, do the development there, and merge the results into the integration environment after that.

Does it mean that all development should be done that way? Not really – merging is not always simple, and you might not want to mess with it too much.

Let’s consider a few examples.

a)    We need to add a new field to the account entity and put it on the form and add it to the views

In this scenario, there is no need for a separate organization. A consultant might do this work directly in the integration environment, and, with all likelihood, other consultants won’t be affected by this change

b)    Compared to the previous scenario, what if we also needed to add a plugin / workflow for the business logic?

If that were a new plugin, we might be ok, but there is a catch. Sometimes we need to debug the plugins, we may need to show error message, we may need to add profiler, etc. Assuming somebody else is working with the entity that we are adding a plugin to, these debugging activities can affect their work. Does it mean we need a separate organization?

Possibly, and the good thing is that plugins are relatively easy to merge (source control tools to the rescue), so it’s the least painful merge problem we can get. On the other hand, we may also try to continue working in the integration environment, and we might just add a condition to the plugin so it only fires for our user account or for one particular record. Something like this might be useful:

if(context.UserId != <your user id>) return;

Or, if it were a workflow, we might just check if it’s running against a specific test record.

You would just need to remember to remove all those special conditions once everything is ready.

c)    Now what if there already were a plugin and we would want to update that plugin instead of creating a new one?

If that plugin were only touching the entities which nobody else would be working on for now, it might still be ok to do development directly in the integration environment.

Chances are, you might want to do this in a separate organization (if you can) in case there is any level of development interference, since, when updating an existing plugin, it’s very likely development and debugging could not be isolated from the existing functionality when done directly in the integration environment.

Another way to work around this problem would be to plan carefully. Split the work between consultants in such a way that there is no interference, and your problem is solved. Can you split the work that way? Possibly not all the time, but, quite often, you should be able to.

d)    What if you wanted to delete an attribute?

That can be a tricky scenario. There might be data stored in that attribute. If you delete the attribute, you will lose the data, and you will lose the audit history, too. That might or might not be an issue. If it is, it might be better to keep the attribute there and mark it somehow instead (add “x” to the display name, for example. That will move the attribute down on the list of attributes when using the advanced find, and, also, you will know that it’s marked for deletion).

In all those scenarios, we can change the complexity of the development process by adjusting the number of dev organizations and, also, by re-arranging the work so that different consultants do not interfere with each other since the dependency between those is more or less like this:

 

This puts more emphasis on the planning project managers/team leads should be doing, and, also, on the overall communication.. Even if it’s not been planned properly, different team members should, usually, be able to re-arrange their work to avoid interference.

Now what I wanted to illustrate above is that we may not have to decide if we’ll be using one organization per developer or if it’ll be a single dev environment at the start of the project. Instead, creating a dedicated dev organization might be considered just one of those tools we should be using in some situations. Even more, it might not be an organization per developer –it might make more sense to have an organization per new feature/set of related features.

3.    Solution components

When you do have multiple organizations, and when you need to bring over the changes from your dev organization to the integration environment (be it a dedicated integration, or production), solution components is what you can use to package those changes:

Add exactly what you need, then export your solution from dev and import to the target organization. Basically, this is the way of not having to do a merge when bringing over the changes. If it’s all been planned carefully, there is a good chance you will only have to bring over those components that nobody else has worked on while you were working on them, so there might be no need for merge at all.

4.    The source control

If you are a developer, you probably can’t imagine your life without a source control solution. It’s your source code repository, it’s your history tracking, it’s your team works, it’s your merge tool, it’s your automated builds and deployments..

Yet if you are a client looking for a CRM tool, source control might be one of the last things on your mind. And, if you are a product team behind Dynamics, I’m guessing you are focusing more on delivering what the clients expect and what they need. Hence, as useful as it is when it comes to development in general, source control integration is missing in Dynamics.

There are some things you can still put into the source control:

–    Custom code for the plugins/workflows/custom actions
–    Web resources

There are, also, tools that may help you automate the process of deploying those components to Dynamics directly from the Visual studio:
https://marketplace.visualstudio.com/items?itemName=DynamicsCRMPG.MicrosoftDynamicsCRMDeveloperToolkit
You don’t have to use the development toolkit, though.
Where the disconnect starts is when it turns out not everything can go to the source control. How do you put an entity form in the source control, for example?

There is a difference between plugin assemblies and entity forms, though. Dynamics does not store plugin assembly’s source code, but it does store form definitions. So, in a way, Dynamics itself is the source control for some of the components. It’s just a much reduced implementation of the source control since there is no versioning and/or history.

Tracking the history for those changes may end up being a manual effort. Literally you may need to create an entity in Dynamics where every developer would describe the changes they are making, why they are making them, and any related change requests/bug numbers. You may use a shared spreadsheet on the Sharepoint site, for example, to do the same. This will create some overhead, so you will have to keep encouraging the practice, but it will all pay off eventually. Sooner or later, you will run into a bug or a feature, and you’ll need to trace the history of that bug/feature. The kind of log I just mentioned will be one of the few tools you’ll have at that moment.

5.    SolutionPackager tool

It is an interesting concept, and it’s a good tool:
https://msdn.microsoft.com/en-us/library/jj602987.aspx

The idea is that you can split Dynamics solutions into subcomponents, store those subcomponents in separate files, and, then, put those files in the source control.

Does it help with the source control? Yes and no.

Yes, we can store everything in the source control now. But what do we do with those individual XML files? There is no editor which allows us to update form XML directly in the Visual Studio, for example. Which means we can only make those changes in Dynamics, then use the solution packager to create an XML.. then try merging that XML with the original file automatically or manually.

That last part looks a bit fishy – to start with, we are not supposed to mess with the customizations.xml so much:
https://msdn.microsoft.com/en-us/library/gg328486.aspx

But, once we start updating those individual files manually, we will actually be updating customziations.xml (which is what all individual files will be packaged into). Whether it will work, when it will break, and, in general, how convenient it is to edit form xml (and other xml) in the text editor, whether you can safely use checked in files to roll back the changes.. It’s not to say that SolutionPackager should not be used on the projects – it’s just questionable whether it’s that helpful.

6.    My scenario

In either case, it seems this is what normally works best for me:
 

Basically, I don’t mind making changes in the integration environment directly – why not.. just think of it as of the Source Control for all your code components, and keep in mind that some of the stuff the source control would do for you automatically have to be done manually in this case.

It may make sense to take a nightly backup of the solution file in this scenario just so that you have something to fall back to when you need.

Then, once everything is ready in the integration environment, you can push it to the higher environments using a solution file.

7.    How do you “delete” an attribute?

So how do you handle “deletes”? A few year ago I would say don’t delete any attributes until at least a few months(or years) have passed since you have hidden them from the forms/views. People change their minds all the time, and your system stakeholders are people, too. But it probably depends on the situation.

Anyway, when using unmanaged solutions, one approach to it would be:

–    Never delete from the lower environment till you have deleted the attributes from the higher ones (so, do it in production first)
–    To start with, mark the attributes being deleted – for example, use “x” prefix for the display name. Remove such attributes from the views, forms, workflows, plugins, scripts, business rules, etc. Check the dependencies, make sure none are left. Do it in the integration environment first.
–    Move those changes to production.
–    Once you get those x-s in production, decide when you want to delete them, and if you want to do it at all. Deletions are final, and you may need to keep all attributes for the possible data audits.
–    If, eventually, you will decide to delete such attributes, do it in production first. You should be able to do it easily since such attributes would have no dependencies anymore.  Once it’s done in production, do the same in the lower environments. Don’t forget to update your solution log file.

8.    And what about testing?

That’s another painful area. It’s almost impossible to test Dynamics solutions from start to end for every new “release” manually, and, yet, it’s extremely easy to break something by changing the business logic in the plugins or by modifying the security roles, or even by removing a value from the option set.

Unless you are ready to mix UAT and production so that your production environment is what you are sort of using for testing, and unless you have unlimited supply of testes on the project, you may have start thinking of the automation.

There are different strategies – you might be using unit tests with fake context, for example, but that’s not an equivalent of user testing.

The only automated test solution I saw so far that would come anywhere close to the actual user testing is this:

https://github.com/Microsoft/EasyRepro

It’s not something just any tester can use.. It’s, actually, more like a test framework for developers, since we have to code test scenarios there. There can be quite a bit of maintenance involved, since, most of the time, you’ll be using xpath expressions to identify HTML elements, and those can change for every new version. Although, since it’s an open source solution, yet the framework itself has being maintained by Microsoft so far, it might be safe enough to use it.

On the other hand, you can easily come up with all sorts of test scenarios – you can start with UI testing/validation, and, then, you can add server-side data validations (since it’s coding anyway.. why not to check how the data looks in Dynamics after you run a specific UI action?)

Most importantly, you can run such tests automatically every night if you wish, and that alone might be a great contribution to the stability of your solutions.

9.    References and must-read

a)    Microsoft Dynamics CRM2011: CRM Solution Lifecycle Management
https://www.microsoft.com/en-ca/download/confirmation.aspx?id=39044

b)    A framework for automating xRM development processes (what I find really interesting about it is the “delete” feature which comes with the so-called extended solutions. Why is it all done with F#, though?)
https://github.com/delegateas/Daxif

c)    Solution Packager tool
https://msdn.microsoft.com/en-us/library/jj602987.aspx

d)    Developer toolkit for Dynamics

https://marketplace.visualstudio.com/items?itemName=DynamicsCRMPG.MicrosoftDynamicsCRMDeveloperToolkit

e)    EasyRepro – automated UI testing for Dynamics

https://github.com/Microsoft/EasyRepro