Capgemini Software Engineering

Devoxx 2024

2024-05-17T00:00:00+01:00

Do queues in the ladies’ toilets signify success at an IT conference? Is a half-hour sit-down worth more than a T shirt? These and other deep questions pondered.

Capgemini’s Cloud Development team are returning from our annual pilgrimage to Devoxx UK, the best and biggest annual IT developer conference in the UK. As is becoming usual, a train strike limited my attendance on the Wednesday, but even in a couple of days there has been much to inspire and mull over. This year, instead of a full booth, Capgemini sponsored a corner filled with super-comfy bean bags, and it was the most popular our area has ever been! As always it felt like a very inclusive conference, with a wide range of age groups and nationalities, and a good balance of genders and races present.

AI Again

As with last year there was a strong theme of AI across many of the conference talks, but this year the emphasis was more on developer involvement - less “Wow look what this can do”, and more “This is how you can build things”. Some angles were difficult to see a production use-case for - for example, Developing Intelligent Apps with Semantic Kernel showcased how with the Semantic Kernel Java API you can allow generative AI access to call your Java functions whenever it considers them appropriate. Having recently taken Capgemini’s “Responsible AI” training course and appreciating that generative AI’s decision making process cannot be considered “correct”, and so you could only use this scenario when it didn’t matter if the AI got the decision to make the Java call wrong, this really limits usage. Deciding when to delete development resources, perhaps? Or rotating a “thought for the day” message? Nothing more important, please! The speaker’s (inevitable) example of roleplaying games seemed safe enough. More AI concerns in Harel Avissar’s talk “The State of Malicious ML Models Attacks”. There are now popular repositories such as HuggingFace where people can share their pre-trained large language models (LLMs), which once trained are, as Harel pointed out, basically deployable units the same as any other software artifact. As such, they are open to attack in the same way as other pipeline artifacts and need to be signed and secured in the same way. This is quite a new field for attacks, but when scanning the LLMs available to download they did find some examples where LLM pickle (.PKL) files, which can run Python scripts when you open them, contained malicious “reverse shell attack” code. Something to be aware of.

AI and Vectors

Quite a few talks focussed on looking at vector databases, commonly used in LLMs, and the algorithms needed to search them. Vector types can represent the relationships between concepts across multiple dimensions. Mary Grygleski’s talk on Friday covered how ChatGPT’s vectors represent 1,536 dimensions. This would mean, for example, it could store 1,536 different contexts for the word “cat”. Handy. But not brand new, and not specific to generative AI - Elasticsearch for example has been using a vector database and the HNSW search algorithm to great effect in its fuzzy searches for years. In fact, as I discovered in my last blog post, the powers of fuzzy search and natural language processing are often more real-world useful than generative AI. You don’t usually want a computer system to come up with its own answer to a question, you want it to go away and find a definitive answer!

Thoughtful Architecture

Conference regular Andrew Harmel-Law was back with what some has said is his best talk yet - “How we Decide”. I didn’t attend as I had recently seen Dave Snowden’s talk on “being human in an age of technology”, in which he went into detail on the biological human process of decision-making - for example, did you know that groups of 5 or less will always reach consensus? Possibly because this is a common family group size, allowing ancient humans to move quickly in family units. And did you know that only 4 in 5 of your decisions are made by the conscious brain, with the other one coming pretty much straight from the body’s senses? Things like pheromones have a big impact on decision-making when face to face. I was still processing this information so skipped Andrew’s talk for a deep dive into container-based IDEs, but I feel I missed out and am looking forward to the video! I did attend Barry O’Reilly’s joyous “An Introduction to Residuality Theory”. Any talk that begins with a Douglas Adams quote and goes on to arm the attending developers with some REALLY DIFFICULT questions to put to enterprise architects (especially difficult if you don’t allow them to use the word “magic”) is always going to go down well, and O’Reilly went on to confirm my suspicions that the way we approach software architecture is less science and more art. He pointed out that our architecture diagrams don’t capture time, change or uncertainty - hugely important factors in software engineering - and how good software architects must be really comfortable with the concept of uncertainty. He had lots of great quotes about how Agile is a reaction to the realisation that requirements don’t work for complex systems, but that using Agile and reacting to change when problems arise leads to flaky architectures. His revelation, and the topic of his Ph.D thesis, is that software architecture could be a science, if we use methods from complexity theory. He went on to outline some really excellent ideas about how to robustify your candidate architecture using random simulation and ideas based on Kauffman’s networks and attractors. Plus, what to do if fire-breathing lizards should happen to climb out of the Thames. Forewarned is forearmed!

More microservices

Microservices haven’t gone away from Devoxx, and this year there were a couple of talks debating / re-debating the monolith v microservice argument. I especially enjoyed Chris Simon’s Modular Monoliths & Microservices - A 4+1 View Continuum in which he advised us all not to bother with the debate as both terms were becoming meaningless! Even a monolith is probably a distributed system. And if you view your client browsers as a scaled-out cluster of your application front end, which you probably should, even the simplest static website becomes a vast distributed network. He recommended moving away from the terms microservice / monolith and instead directly using terms such as “process” (a thing you can start and stop), and “node” for distributed compute, and teams and repositories for code ownership. He talked about the power of Domain Driven Design (DDD) when building up the logical view of your system, and how mapping processes onto bounded contexts can give you a candidate physical architecture, although don’t shy away from multiple processes in a bounded context if, for example, there are asynchronous steps between multiple processes (i.e. a queue). He also gave the quote of the conference when he said people often found they had a service and “distributed the wazoo out of it and we don’t know why”. He talked about the development view of an architecture and how your code repository should always align with the logical view. If this doesn’t happen you end up with inter team coupling - behaviours such as “Scrum of Scrums” mean that it is becoming very hard to get decisions made and is a clue that your development view isn’t in alignment. He also talked about how we should reconsider our assumptions on the physical view of the architecture - we often jump to the conclusion that a microservice needs its own infrastructure but this may not be the case, and we often assume that a repository must be deployed as a monolith when in fact you can use path filters on your monorepo to deploy separate sections at different times.

Fear of Rust

For something new, I went to see Ayoub Alouane talk about “My discovery of Rust: Why is it a Game Changer?” where he showed an example of Discord’s “Message Marker” app showing regular spikes in CPU that corresponded with garbage collection. The corresponding Rust application had no such spikes. Why? Rust, announces Ayoub, does its memory management at compile time! He went on to demonstrate with a simple loop application how, for each variable, you had to specify the memory details of the variable:

Is it mutable?
Is it usable in multiple threads?
Is it lockable?
Should you clone it into threads?

All this can be expressed through the syntax of the language. A bit of a pain to learn, but well worth it. A lot of end-of-life C applications are being migrated to Rust, and after this talk I can understand why. He gave another example of rewriting a node.js app in Rust and using 95% less memory and 75% less CPU! OK, game-changing point taken! This talk of application efficiency fit well with our own Shirley Tarboton’s talk on Decarbonising Code. She gave some examples of easy-to-implement coding practices that can make your application consume fewer resources. It was a well-received talk in the huge auditorium theatre - quite a feat to get people to attend during lunch!

Crashing and burning

Last thing Thursday I popped into a talk called Mayday Mark 2! More Software Lessons From Aviation Disasters.. As speaker Adele Carpenter pointed out, the Venn diagram of software engineers and aviation geeks is, “basically a circle”. Previous aviation disaster talks I’ve been to focus on ways of working we can both benefit from - for example, focussing on psychological safety, failing fast and failing openly. Adele’s talk, however, was a rather harrowing blow-by-blow account of some major air crashes from the past few decades. There were some important lessons about how much information humans can cope with, how humans react unexpectedly in stressful situations, the importance of familiarity and expectations in UI design, but overall it was a rather sobering affair. Which is just as well, since we all poured straight out of the talk and into the local pub for the famous Devoxx party, complete with IT-branded beers. Cheers for another year!

Continuous Change

On Friday I went to a talk very relevant to my current project - Chris Simon’s “Winning at Continuous Deployment with the Expand/Contract Pattern”. My project has a fairly typical Continuous Integration pipeline, but we still do production releases via manual intervention on a business schedule. One of the reasons is schema changes which cause backward incompatibility between services - this can cause outages if our server-side (message recipient side) services are updated before the client side (message sender) services - because even if they manage to process the old-style message, they may send a response the new client isn’t expecting. So whilst we have eventual consistency, a few requests may get lost along the way during the service restart. The Expand/Contract pattern addresses this by putting the onus on developers to create an “interim” server-side application which can accept both old and new client messages, and sends both old and new response formats. So if you used to have an application that took a username and returned OK like this

{
 "name": "Sarah Saunders"
}

{
 "response": "OK"
}

and the new version would take first name / last name and return OK and an ID:

{ "name": {
 "first": "Sarah",
 "last": "Saunders"
}}

{
 "id": "2134-ker-438052u"
}

In this scenario, the “interim” application would accept both inputs, and would return an output like this:

{
 "response": "OK",
 "id": "2134-ker-438052u"
}

This should allow client applications to talk to the server both before and after they are migrated to the new version. Then, once all clients are migrated, the “Contract” stage of the release occurs and the server can be moved to only accept/respond in a V2 style.

I really like this pattern and I don’t think developers would mind writing the extra code for the interim release. It does spell out though, that deployment must be a consideration right from the design stage of an application, and Chris did raise this point at the start - mentioning his other talk goes into more detail about how Test Driven Development (TDD) helps this by defining strong contractual interfaces between components, and is a strong enabler for continuous deployment. I have long been a strong advocate of TDD, and here is yet another reason why getting ChatGPT to write your tests for you is no substitute at all for proper TDD!

In conclusion, a really great conference for affirming that we (Capgemini Cloud Development team) are on the right lines with the software and architectures we are currently building for our clients, our use and understanding of AI aligns well with the global community (or is sometimes a little bit ahead) and we have lots of great lessons to learn to improve our architecture and deployment practices even further. Looking forward to the next year.

Devoxx 2024 was originally published by Capgemini at Capgemini Software Engineering on May 17, 2024.

Micro Frontends: an Introduction

2021-10-08T00:00:00+01:00

Microservices & Distributed Systems in General

The growth in popularity of distributed systems is not without reason. Organisations have found that the separation of workstreams along domains / business capabilities has provided teams with a greater degree of product ownership. There is an increase in the overall complexity of the aggregated services of course, as interactions between services have to be agreed (often between multiple teams), but the advantages of independently deployable & scalable components and greater team autonomy should make microservices an attractive option for most projects.

There is a caveat to this however: the term “microservice” is pretty much exclusively used to refer to backend web services. These are services that are usually obscured behind an API gateway and a user interface of some kind. There have been lots of weird and wonderful innovations to improve the way we design and build distributed systems, but that effort is primarily focused around improving consistency and availability for your backend services (not to mention the countless frameworks, plugins and tools to marry with whatever patterns you decide to go with for your system). This isn’t exactly surprising, if you’re going for as close to perfect as possible for your system’s consistency and availability then you’re going to focus on the services that are actually performing the tasks required of the system.

User & Developer Experiences

If we move out of the system’s boundary and look at how our users see it from an external view, we’ll find that they don’t (or rather they really shouldn’t) see anything that gives away its nature or underlying architecture. Users expect a seamless experience when they interact with an application, aesthetically as well as functionally. If your interface seems to change its design system in places, that’s going to make your system look like a patchwork of different bits of software and the illusion of a single consistent application is broken. If your interface doesn’t have clear and unified paths for performing actions, then your users may not be able to use it and you’ve suddenly got a very serious problem.

These risks, combined with the fact that interfaces like web apps are essentially just bundles of static assets that users have access to, has led to the default architectural option being some flavour of monolith for frontend applications. Now I’m sure I don’t have to go through the effort of explaining where monolithic applications fall short, those of you who haven’t had the pleasure of building or maintaining one will undoubtedly have heard tales from those who have. However, it’s worth touching on some techniques that have been used to attempt to distribute work on a monolith between different teams.

A module-based approach can seem like a good idea on first consideration, as you can neatly divide up an application and give ownership of them to various teams. What should also be considered is the aggregation of these modules into the final artifact. At some point, these components are going to have to be bundled together, tested, packaged into a deliverable and deployed. The coupling between these modules is usually tight, which increases the likelihood of changes in one component impacting another. If this is the case, then you’re not making the most of a distributed system since your teams will still have to negotiate when making changes that affect the interfaces between their components.

Now that they’ve been put into the appropriate context, we can finally start talking about micro frontends.

Micro Frontends: a New Alternative

Let’s start with a (fairly vague) definition:

A micro frontend is a semi-independent component that can be independently deployed, and dynamically integrated into, a user interface.

This doesn’t really do much for us, so let’s dig into it a bit. What we’re basically talking about is an extension of the thinking that brought microservices into being - what if we cut up our frontend along some meaningful boundaries and developed them separately? You can split the work between teams and let them build their own components in their own way, which is the first advantage this approach brings. Each team doesn’t have to negotiate with every other team to be able to use their preferred tools, since they own the development and deployment aspects of their product. You could have teams using completely different JS libraries, built using different CI tools and deployed to different platforms, and you can still integrate their work together to form a single seamless frontend. This approach will be appreciated by the teams, as they’re freed from the need to agree with all other teams on what tools they should all use.

Since your teams can all go off and deliver the functionality they’re responsible for, you can also bring in more contributors without impacting developer experience (please note I’m emphasizing the word “can”, as simply throwing bodies haphazardly at a project tends to make things worse instead of better). As long as your frontend is decomposed sensibly, and teams are working together to communicate across their component boundaries properly then you shouldn’t have any problems.

NOTE: At this point it’s worth mentioning that micro frontends are still evolving as a concept and there are lots of ways to use them, so take some of this with a pinch of salt. You should apply these methods in a way that works best for your project, but I can speak from experience that the techniques mentioned in this article have worked for my past projects.

If we take a standard web application and start cutting it up, we can see some clear potential micro frontends:

An ecommerce application, with highlighted sections that can be developed as micro frontends

Here we can see that there are micro frontends to be made for the navigation bar at the top of the page, the cart summary popup, and the store item components. In theory you could have each of these developed and owned by different teams, and integrated together into a single application.

Integration

It’s all well and good to develop parts of an application in isolation, but at some point, you’re going to have to stitch everything together. For microservices this is usually achieved with an API gateway, which serves as the entry point into a system and sometimes takes care of cross-cutting concerns such as authentication pathways. Micro frontends also have a single entry point when a user accesses them, usually referred to as a container or root component. When the application is loaded up this is what is given to the browser, and within it is the code required to load up other components from different micro frontends.

The difference between micro frontends and module-based monoliths should be made clear here; the container application will dynamically load components into the browser DOM from remote sources when they are needed, the initial bundles of JS and HTML that are returned from the user’s initial request do not contain any components belonging to other micro frontends. Instead of this, the root component will contain URLs to various micro frontend resources that are retrieved separately (these could be hosted at a different address, built by a different team using different technologies, and deployed using a different cloud services provider). This gives teams the advantage of being able to deploy a new version of a micro frontend, and have this new version become immediately available through the root component without having to pull new versions into any other components and build them again. You will still need to make sure that any contracts you have established between components are still being respected of course, but you can use the usual techniques for dealing with breaking changes when required.

State & Communication

Building an application with completely isolated components would be a very straightforward thing to do, but I’m sure we’re all aware that it is rarely a scenario that comes up on actual projects. It’s usually inevitable that components will eventually need to communicate with each other, responding to either the user or another system’s actions. Now one way to tackle this would be by following the unidirectional data flow pattern, utilizing libraries like Redux to manage state. This pattern is useful for decoupling UI components from state modifications, but the concept of a central state store presents some problems for us in an application composed of micro frontends.

It would be tempting to keep a state store in the root component and have this passed into micro frontends when they are loaded. This would give each micro frontend access to the same state store, allowing them to share data with each other and update themselves when any changes to the state object are made. The problem with this is that you are adding a dependency on the type of state management library the root component is using, which goes against the idea that each micro frontend should be independent. Micro frontends should be able to differ in their implementation without affecting other components in the final application, so restricting them in this way should be avoided wherever possible.

We can allow each micro frontend to manage its own internal state, but there is still the problem of communication. It could be tempting to define interfaces that are implemented in each micro frontend and made available to any component in the application that needs to use them, but this can create tightly coupled micro frontends which makes things difficult to maintain and augment. It is here that we could get some benefit from thinking about this in a different way.

Instead of thinking about side-effects of actions as something that a component applies to another component through a function call, we can think about an action as an event that is broadcasted via a message broker. By doing this we can keep things loosely coupled, allowing easy integration with other components through subscribing to particular message channels or topics. For example, if we had a shopping application with an “add to cart” button, this button could trigger an event that all interested components (a cart contents component for example) could subscribe to. We can use a number of tools (my current preferred choice being Postal.js) to achieve this, adding relevant data (item IDs etc.) into the body of the event if required.

An advantage of this event-based approach is that you can document your APIs to be used by other teams when integrating with your micro frontend. Tools like AsyncAPI excel at this, giving you a neat, unified definition of all the channels and events that your component(s) will potentially emit, and what they signify. You can also version your events as you would for event-driven systems, allowing for gradual phasing out of old event processing as your system matures and changes. Your API specification can also include information on the build tools you are currently using to export your micro frontend, and the dependencies that it is set up to share (shared dependency resolution is a feature of a number of frameworks used to integrate micro frontends, and would be very important for teams to see when preparing to integrate with your components).

Potential Problems

Building a micro frontend-based application has many advantages, but there are also some potential issues that you should be aware of so your team are spared some painful problems in the future.

Integration technology alignment: you will have to ensure that all teams working on micro frontends are working with the same technologies for exporting their components, as the root application will need to load them all in. There are various options out there, Webpack’s Module Federation plugin and single-spa to name a couple. Module Federation is very flexible, allowing you fine-grained controls over things like shared dependencies. However, single-spa is very quick to get set up and comes with very useful run configurations out of the box (to assist with local development and running micro frontends in isolation). I have had success in the past with the Webpack solution, but you should choose the tool that’s going to work best for your team(s).
Performance: as your development teams are given the freedom to choose the libraries that they use in their micro frontends, there is a risk that your application will become bloated. There are patterns and techniques you can introduce to mitigate this, both technical and organisational. If you have dependencies that are used in multiple micro frontends, you can configure whatever tool you’re using for builds to use dependencies that are shared by the root component. For example, if you’re using React in a number of micro frontends then you can add this in the root as a shared dependency in your Webpack or single-spa config, and then any micro frontend that uses React will be able to load without having a duplicate version of React bundled into its own JS files. You can also collaborate between your various teams to select various preferred libraries, which will limit the amount of different dependencies being loaded into the final application.
Debugging: As you can imagine, debugging an application with lots of event-driven micro frontends can get quite complicated, especially when initially integrating new components into an application. If you’re using an event bus like Postal.js, then using a plugin like postal.diagnostics can help you track events passing over the boundaries separating your micro frontends. There can also be some difficulties when working with your build tool to correctly bundle and load micro frontends into an application. This can be very difficult to debug when you’re having loading issues, so I’d advise creating a scaffold with a bare-bones implementation for exporting a micro frontend using something like Yeoman. With this you can tailor your generator to suit the functionality that your team may need for preparing a new component for integration, decreasing the possibility of running into integration issues when introducing a new micro frontend into your application.
Styling clashes: since you will be loading components built by different teams into the same DOM, you’ll need to put measures in place to avoid CSS class clashes. This can be solved by agreeing to use prefixes in all your CSS classes, or using something like CSS modules.

Conclusion

Micro frontends have a lot of potential if used correctly. It is possible to build large, complex applications by allowing different teams to own part of the web application, but you should take care before you jump straight into using them. As with most patterns and techniques, preparation is key. As long as your teams are communicating properly, and cross-cutting concerns are being handled in a way that is understood by all involved then you shouldn’t run into any problems (at least none that can’t be solved in a relatively straightforward manner).

Micro Frontends: an Introduction was originally published by Capgemini at Capgemini Software Engineering on October 08, 2021.

Enterprise Architecture DocOps

2021-06-17T00:00:00+01:00

Introduction

The purpose of Enterprise Architecture is to optimize across the enterprise the often fragmented legacy of processes (both manual and automated) into an integrated environment that is responsive to change and supportive of the delivery of the business strategy. The Open Group Architecture Framework, Part I: Introduction

One of the tasks of an Enterprise Architect (EA) is to produce a set of artifacts that collectively form the architecture documentation. Such documentation constitutes the main input to Architecture Governance (AG).

Architecture Governance is an approach, a series of processes, a cultural orientation, and set of owned responsibilities that ensure the integrity and effectiveness of the organization’s architectures. The Open Group Architecture Framework, Part VI: Architecture Capability Framework

In the absence of general consensus on architecture documentation’s format, structure and content, a number of problems have made the subject somehow vexed within the EA’s and software development communities, leading to well known issues like artifacts that are:

outdated, created some time in the past, the original authors unknown or moved on
crafted without clear purpose, by different people with insufficient coordination
overwhelming in volume, made of information scattered in various files or wiki pages without structure
painful to write and maintain because goals and tools to use are undefined or unclear

Special discomfort is provided by diagram maintenance, where energy is spent during inception phase to create shapes, connections, fitting layout…, which soon enough becomes wasted, as new updates are required and a lot of effort needs to be spent again to get the artifact to a reasonable level of quality. Not to mention versioning: the delta between different versions of a diagram is not immediately visible and can be partially missed if not inspected with care.

On top of that, in the context of governance, documentation review and approval processes vary a lot, often not fully defined and lacking formalities, e.g. based on files exchanged and reviews happening via email.

docToolchain

Documentation as Code (docs-as-code) refers to the philosophy advising authoring documentation with the same tools as code. Text-based version control systems, like Git, simplify and bring formality to review and approval processes via workflows and pull requests, enabling then easier and clearer paths to documentation finalisation.

AsciiDoc is a markup language, embracing the docs-as-code approach and primarily conceived to write technical documentation. Thanks to an AsciiDoc processor, like Asciidoctor the language produces a variety of output formats, such as HTML and PDF.

Similarly to docs-as-code, diagrams-as-code adopts text to represent and produce diagrams, with similar benefits.

Asciidoctor Diagram is a set of Asciidoctor extensions that empower the author to add text-described diagrams to an AsciiDoc document. Each extension runs the diagram processor to generate an SVG, PNG, or TXT file from the input text. The generated file is then inserted into the converted document. There are nearly thirty diagram generators supported and amongst them the popular PlantUML, which facilitates designing a number of diagram types, e.g.: sequence, class, state, timing, JSON, YAML, Gantt and many others. A few rendering examples below (source code in Appendix):

Sequence type

Class type

JSON type

arc42 is a template for documenting software and system architectures, whose golden master is formatted in AsciiDoc and which is publicly available. It is segmented into twelve sections, each containing help, divided into contents, motivation and form. Real-world examples are also available.

Putting it all together is docToolchain, a set of scripts that automate the steps of exporting AsciiDoc documents (including arc42-based ones) and rendering diagrams, all to the chosen target format (e.g. HTML).

DocOps with Azure DevOps

DocOps applies to the creation, management, and release of documentation, similarly to those applied to source code by DevOps: it is a set of practices automating and integrating the process of developing documentation across engineering, product, support, and technical writing teams.

Starting with docToolchain, in order to fully comply with the DocOps approach, the missing piece is the automatic deployment of the documentation, once the approval is granted (in the form of an approved pull request).

To cover that “last mile”, I decided to host the documentation on Azure DevOps Repos and implement a pipeline in Azure DevOps Pipelines that, after being triggered by an update to the documentation repository master branch, builds an HTML page with text and diagrams and deploys to a website. For the purpose of this exercise I decided to use a static website hosted in Azure Storage as the destination (a lot of other types are available, selection depending on requirements).

(The PlantUML diagram definition for the above is in Appendix.)

Here is the definition of the pipeline in YAML:

variables:
  documentationRoot: #subpath to the folder where arc42-template.adoc is located,
                     #e.g. after arc42 template download and unzip: arc42-template-EN-withhelp-asciidoc

resources:
  repositories:
  - repository: docToolchain
    type: github
    endpoint: #name of the Azure DevOps connection to github
    name: docToolchain/docToolchain
 
trigger:
- master

pool:
  vmImage: ubuntu-latest

steps:
- checkout: self
- checkout: docToolchain
  submodules: recursive
- script: sudo apt install graphviz #dependency of some PlantUML diagram types
  displayName: 'Install graphviz'
- script: |
    cd docToolchain
    rm -rf .git
    rm -rf resources/asciidoctor-reveal.js/.git
    rm -rf resources/reveal.js/.git
    ./gradlew -b init.gradle initExisting -PnewDocDir="$(Build.SourcesDirectory)/$(documentationRoot)"
    ./bin/doctoolchain "$(Build.SourcesDirectory)/$(documentationRoot)" generateHTML
  displayName: Generate HTML

- task: AzureCLI@1
  displayName: Azure File Copy to Storage
  inputs:
    azureSubscription: #name of the Azure DevOps connection to the Azure Subscription 
                       #where the storage container is hosted
    scriptLocation: inlineScript
    inlineScript: |
      az storage blob upload-batch \
        --destination \$web \
        --account-name "myStorageAccount" \
        --source "$(Build.SourcesDirectory)/$(documentationRoot)/build/html5"

Conclusion

I like working with AsciiDoc for personal and shared documentation: advantages over Markdown, the more widely adopted alternative, are extensibility and a unique flavour, compared to the many existing for Markdown, which make the language difficult to port between environments. I also appreciate the large range of diagramming tools supported.

I believe diagrams-as-code is a handy approach for versioning images together with the document and enabling exact and fast delta highlighting. The default styling, especially in PlantUML, is quite basic, but can be customised, if needed. Adopting the default style though, presents the advantage of consistency between documents and authors, which gives the reader faster grasping of the concepts depicted in the diagrams.

I found arc42 a good starting point as a template, though I feel it is missing the next level of detail, e.g.: sections related to the Logical Information Model (objects, stores and flows) and a Security view.

Finally, I particularly appreciate the workflow achieved with text-only documentation, source control and CI/CD: the easy collaboration, the approval process, the clear versioning, the history tracking that comes with it, the evidence of which version is current, and the auto-deployment.

Appendix: PlantUML diagrams source code

Sequence type

@startuml
actor User
User -> "Authentication Provider" as Auth : Authenticate with Client ID + Client Secret
activate Auth
Auth ->  Auth : Validate Client ID + Client Secret
Auth --> User : Access Token
User -> "Web Service" as Service : Request Data with Access Token
Service --> User : Response
@enduml

Class type

@startuml
class Animal {
  age
  gender
  isMammal()
  +feed()
}

class Duck {
  featherColour
  beak
  +swim()
  +quack()
}

class Lion {
  maneColour
  +roar()
  +chase()
}

class Beak {
  colour
  length
  +open()
  +close()
}

Animal <|- Duck
Animal <|-- Lion

Duck *- Beak
@enduml

JSON type

@startjson
{
  "name":"J. K. Rowling",
  "born":"31 July 1965",
  "genre":"Fantasy",
  "country":"United Kingdom",
  "occupation":"author",
  "books":[
    {
      "title":"Harry Potter and the Philosopher's Stone",
      "yearPublished":1997,
      "pages":223
    },
    {
      "title":"Harry Potter and the Chamber of Secrets",
      "yearPublished":1998,
      "pages":251
    }
  ]
}
@endjson

DocOps workflow

@startuml
!define AzurePuml https://raw.githubusercontent.com/plantuml-stdlib/Azure-PlantUML/master/dist
!includeurl AzurePuml/AzureCommon.puml
!includeurl AzurePuml/Storage/AzureBlobStorage.puml
!includeurl AzurePuml/Identity/AzureActiveDirectory.puml
!includeurl AzurePuml/DevOps/AzurePipelines.puml
!includeurl AzurePuml/DevOps/AzureRepos.puml
!includeurl AzurePuml/General/Azure.puml

skinparam linetype polyline
skinparam linetype ortho


file "arc42 template" as arc42
Azure(azure, "Cloud computing", "Subscription", ) {
  AzureBlobStorage(blob, "Storage Account", "$web Container",) {
    file HTML
  }
  AzurePipelines(pipeline, "Build + release", "CI/CD",) {
    [GraphViz]
    component docToolchain {
      [Asciidoctor]
      component "Asciidoctor Diagram" {
        [PlantUML processor] as PlantUML
        [Other processors, e.g. Mermaid] as otherProcessors
      }
      [Other generators, e.g. PDF]
      [HTML generator] as HTMLgenerator
    }
  }
  AzureRepos(repo, "Architecture documentation", "Repository", ) {
    file "Diagrams, format: e.g. PlantUML"
    file "Text documents, format: AsciiDoc"
  }
  AzureActiveDirectory(ad, "Authentication + authorisation", "AD",)
}

repo ---> docToolchain
arc42 <|--- repo
HTMLgenerator --> HTML
GraphViz -- PlantUML: < dependency

otherProcessors -[hidden]-> blob
@enduml

Enterprise Architecture DocOps was originally published by Capgemini at Capgemini Software Engineering on June 17, 2021.

The Conservation of Complexity in Software

2019-03-22T00:00:00+00:00

In physics, the law of conservation of energy states that the total energy of an isolated system remains constant—it is said to be conserved over time. Energy can neither be created nor destroyed; rather, it transforms from one form to another. In IT I postulate that there is another similar law that we all should get familiar with:

The conservation of complexity states that the total complexity of an isolated business / IT system remains constant – it is said to be conserved over the lifetime of the solution. Complexity can neither be created nor destroyed; rather, it transforms from one form to another.

If energy is the ability to do work, or the ability to move or elicit change, then complexity (in this context) is the resistance to change in a system. This resistance causes the effort and time to implement a change to increase which in turn increases the cost of sustaining and enhancing a system. How do I come to that conclusion?

Complexity in Architecture

The following diagrams model the way in which system architecture has developed over the last few decades. Each diagram represents the same generic business problem solved through differing architectural styles. Monolithic systems have given way to client/server or tiered designs which have in turn fallen out of favour to be replaced by service architectures whether the SOA or the Microservices kind.

Figure 1

Each progression above can be classed as simpler than its predecessor. Components and then services have been introduced as a means to compartmentalise business logic in order for it be reused or replaced altogether. The drive to Microservices takes this further. It is the manifestation of the Single Responsibility Principle at architectural level. Building a service that does exactly one thing well is much easier than trying to weave that code into a monolith. There are no distractions and it is straightforward to articulate the acceptance criteria or what done looks like. However, one service does not make a solution, so what is the impact to the overall system complexity?

Figure 2

Layering on inter-component or inter-service interaction on to the client/server and Microservices models above highlights that the complexity has shifted to the networks and communication channels between the components that make up the system. It seems that the client and server components are complex to build but simple to connect, whereas Microservices are simple(r) to build but more complex to interconnect.

Building communication networks between components adds another layer of complexity. Nodes in the network need wiring together and managing. Security becomes more of a concern as the traffic travelling on the network needs protecting. More network hardware is introduced, and someone has to manage it. It may be easier to test each component individually but how do you know that the system in its entirety is working? When things go wrong how do you pinpoint the cause?

The complexity has simply been relocated from the software and code into the network and solution management.

Complexity in People

People have roles to play in complexity, after all in many software architectures, people are a fundamental part of the system.

In the early days of automating business processes using computers, the software often played the part of a glorified filing cabinet. Records are accessed in the system, they are reviewed, changed if necessary and then pushed back. The users and the software are working together to achieve some business activity, often with users holding relatively complex processes in their heads. Sometimes the people using the system act as an integration layer. Data is read from one system and keyed in manually to another.

As activities are automated and the burden on people is reduced the complexity moves into the system. New user interface styles are designed and built so users can be more efficient. Workflow systems are introduced that allow humans and systems to communicate more effectively.

In essence complexity has been moved into the software to make life easier for the users. The complexity of the complete system has not changed.

Complexity in Phasing

Even in a world where DevOps is gaining popularity, it is still typical for software to be born in a large-scale delivery project, at the end of which it is transitioned into support where it is run until it is no longer required. In my experience these large delivery projects are where large-scale investment takes place and where attention is focused. However even the best plans need to be changed and scope is often reduced. Business functionality is prioritised over operational requirements and before you know it the software is in support, but it is complex to operate.

The complexity of the system has not changed. We have simplified the delivery timeline, but all the complexity has moved to the support team. More people are needed to run the system, more telemetry is required to understand what is going on and the solution is more expensive to operate.

In Conclusion

As people responsible for the successful delivery of software we need to be aware of the consequences of our choices. Our jobs can be pressurised and it’s natural to try to make life simpler but the choices we make can have a wide impact. When we consider the complexity of the entire system, we can assert whether our simplifications are positive. We might be making life worse for the people who have to run our software or for the people using our software, or even our future selfs when we are called back to fix our software.

The Conservation of Complexity in Software was originally published by Capgemini at Capgemini Software Engineering on March 22, 2019.

Combining OAuth and JWT to gain performance improvements

2018-07-13T00:00:00+01:00

For many years Simple Object Access Protocol (SOAP) was the standard approach for communicating with remote services, often via HTTP. The landscape has changed significantly in recent years with the increase in the adoption of Representational State Transfer (REST) APIs. There are still a number of use cases that suit SOAP, for example where stateful operations are required.

One of the six guiding architectural principles of REST is statelessness. Every API request from the client to a server must contain all the necessary information necessary to serve the request. The server maintains neither state nor context. How do we then authorise access to protected REST APIs? Say hello to OAuth and JSON Web Token (JWT). OAuth and JWT are two of the most widely used token frameworks or standards for authorising access to REST APIs. In this blog post I consider how both OAuth and JWT can be combined to gain performance improvements.

OAuth enables an application to obtain limited access to an HTTP service. While JWT is a compact, URL-safe means of representing claims to be transferred between two parties. OAuth has a number of grant types. So whenever I refer to OAuth in this blog post, I am referring to the OAuth 2.0 Resource Owner Password Credentials Grant type. A user is required to authenticate or login to obtain a token. A typical authentication flow is shown in the sequence diagram below.

A user will enter their username and password via a client (which could be a mobile device or PC), and at the end of the authentication process the user will be supplied with a token. The client will then include the token with every subsequent API request to a resource server (like the User server). To compare what the authorisation flow for both OAuth and JWT will look like, let’s consider an example where we make an API request to GET the authenticated user.

The OAuth flow to GET the authenticated user whose ID is 123 will typically look like the sequence diagram below.

While, the JWT flow to GET the authenticated user whose ID is 123 will typically look like the sequence diagram below.

The JWT implementation is less chatty and more performant compared to OAuth. This is because JWT enables a resource server to verify the token locally. In its compact form, JWT consist of three parts: the header, payload and signature. The signature is the result of signing the base64Url encoded header and the base64Url encoded payload with a key. The resource server uses the signature to verify that the token has not been tampered with.

The JWT payload contains the claims. The claims are statements about an entity (typically, the user excluding private information of course) and additional metadata like the expiry time. JWT however has a drawback in that once it has been issued it will allow its holder to gain access to a resource server until the expiry time is lapsed. It looks like we need a way to revoke JWTs. So let’s go back to OAuth.

OAuth is chattier compared to JWT. This is because OAuth requires the Auth server to verify the validity of the token and the Auth server in turn relies on the information it has stored in a database to make this judgement. OAuth however does have an advantage over JWT in that tokens can be easily revoked. This is particularly a good feature if instant access revocation is desired. The basic OAuth token response is shown below.

{
    "access_token": "foo",
    "token_type": "example",
    "expires_in": 3600,
    "refresh_token": "bar"
}

The relevant attributes are described in the table below.

Attribute	Description
access_token	The short-lived token that is typically valid for about an hour.
expires_in	The lifetime of the access token in seconds.
refresh_token	The long-lived token which are used to obtain new access tokens. Typically valid for a number of days or months.

As shown in the example OAuth flow above, the client will include the access token in every API request to a resource server. The client will then make use of the refresh token to obtain a new access token. How can we benefit from the inherent performance advantages associated with JWT and the limited access capability provided by OAuth? By issuing an OAuth token with JWT in both access token and refresh tokens as depicted below.

{
    "access_token": "aShortLivedJWT",
    "token_type": "example",
    "expires_in": 3600,
    "refresh_token": "aLongLivedJWT"
}

The client will include the short-lived JWT for every call to the resource server, and will make use of the long-lived JWT to obtain a new access token. The short-lived JWT is validated locally. We do however need to keep a record or blacklist of the revoked refresh tokens till they expire. This blacklist will be checked only when the client wishes to refresh the OAuth token. A new access token will not be granted if the refresh token is found in the blacklist. The blacklist should ideally contain refresh tokens associated with users who have logged out and users whose account have been disabled. Although the access token is not immediately revoked, it is meant to be a short-lived token.

Finally, there may be scenarios where this behaviour is not desired but ultimately, it depends on the requirements of the system. This approach enables the resource server to validate the OAuth access token locally and only requires interaction with the Auth server when we need to get a new OAuth access token. It is this reduction in the interaction with the Auth server that gives us the performance improvements.

Combining OAuth and JWT to gain performance improvements was originally published by Capgemini at Capgemini Software Engineering on July 13, 2018.

Techniques to improve Application Design

2016-03-11T00:00:00+00:00

Design is important because it gives an application a vision; a vision that can be shared amongst the team.

It gives the team a direction; a foundation on which to build the code. It is not a means to make the implementation easy but to make it efficient.

There are several ways to design an application. I am by no means an expert but I would like to share my experience that can make application design a little bit easier.

Start small

Start the design with something that gives the team the best possible start with coding. It is not necessary to design the entire application in one go. It’s all about microservices now. But before that it was (it still is) modules, services, interfaces etc. However you decide to break down the application in question, start with the smallest logical part. Start building a mental picture of how this all fits together. Address the functional complexities first, making sure not to lose track of the domain principles. Add the additional but essential features later on. For e.g. you do not want to start the design by worrying about incorporating configuration, logging, monitoring or metrics stuff into your design.

Keep it simple

Simple designs are usually the most powerful, efficient and also the most difficult to conceive.

I once designed a screen which would allow a developer/support person to configure the UI for various modules of the application, a settings screen, if you like. I was just finding my new talent in design, so in all the excitement I started off with the vision that my configuration screen will be totally configurable - Yes, a configurable configuration screen. The intention behind this was to make it utterly flexible for the user which in my case was probably another developer or someone with an IT background. It had asynchronous front end loading and all the hot stuff (at that time). But developing that screen was a nightmare. I managed to finish it but a couple of months down the line, even I couldn’t make head or tail of the configurations. The lesson for me was to keep things simple. It doesn’t have to be the Superman of the design world - the one that Flies, Saves the World and looks nice and handsome doing it.

Delegate the complexity if it is bearable elsewhere. In my case, the user would have been perfectly capable of handling that extra bit of work. Knowing the audience really helps.

A practical design not an ideal one

You have a reasonable design - what next? Here comes the difficult part. Implementing it. This is when the design is put to actual test.

All designs work well on paper. You have to navigate through a myriad of technological challenges before the design actually works. More often than not, you might need to rethink parts of it. Trade-offs are an unavoidable part of designing an application. Sometimes, a better user experience will call for a trade off in the design.

The important thing to keep in mind is not to forget why the decisions were made in the first place. It might have been done in a certain way to represent an important domain concept or parts of it. The challenge is to make a trade off without sacrificing on the things that really matters such as core domain principles, performance etc.

Feedback

Look for constant feedback from the implementation cycle. Keep an open mind and never hesitate to go back to the drawing board if required. Some of the best designs on paper may end up being the least efficient ones. Agile delivery methodologies and some recent architectural patterns such as microservices can help us in this regard. Find out the trip ups early on in the cycle and fix it.

Starting small, as mentioned above, really helps in this regard. I prefer doing my design on a whiteboard because it is easy to accommodate changes, instead of drawing endless number of UML diagrams using tools and redrawing them when a tiny thing changes. It also helps me be disciplined about the scale of the design.

Foresee

I have found in my experience that adding a bit of flexibility in the design really helps. It could as small as having switches to turn things on and off - these are tiny little things that don’t get mentioned in the requirements but really helps when the code rolls out to production. It does need a bit of foresight, but it is important not to get carried away.

The Unbreakable Design - There is no such thing

It is absolutely essential to make the application as fault tolerant as possible. Think about all possible scenarios however unlikely or brief they may be while designing the application. Now, that is not enough. Because in the real world there is always something lurking in the dark corners (a.k.a edge cases), that will break the code. The best possible thing to do is to build a cushion to protect your application when the inevitable happens. Let it break, but make sure it recovers gracefully.

To sum up, while designing an application it is good to follow three A’s:

Accept - that the design is not prefect and strive for constant improvement. It is not a one-off activity, otherwise it runs the risk of becoming stale.
Accommodate changes to the design if it means improvement elsewhere. A rigid design can easily compromise the usability of an application.
Be Agile. Start small and iterate. Be sure to incorporate the feedbacks in the subsequent iterations.

Techniques to improve Application Design was originally published by Capgemini at Capgemini Software Engineering on March 11, 2016.

Is REST Best in a Microservices Architecture?

2015-12-18T00:00:00+00:00

During my journey into microservices, it has become apparent that the majority of online sample/howto posts regarding implementation focus solely on REST as a means for microservices to communicate with each other. Because of this, you could be forgiven for thinking that RESTful microservices are the de facto standard, and the approach to strive for when designing and implementing a microservices-based system. This is not necessarily the case.

REST

The reason why REST based microservices examples are most popular is more than likely due to their simplicity; services communicate directly and synchronously with each other over HTTP, without the need for any additional infrastructure.

As an example consider a system that notifies customers when a particular item is back in stock. This could be implemented via RESTful microservices as so:

An external entity sends an inventory update request to a REST gateway address.
The gateway forwards this request to the Inventory Manager Service.
The Inventory Manager updates the inventory based on the request it receives, and subsequently sends a request to the Back in Stock Notifier.
The Back in Stock Notifier sends a request to the Subscriber Manager, requesting all users that have registered to be notified when the item is back in stock.
It then emails each in turn, by sending an email REST request to the Email Service for each user.
Each service then responds in turn, unwinding back to the gateway and subsequently to the client.

It should be noted that although communication is point-to-point, hard coding services addresses would be a very bad design choice, and goes against the very fundamentals of microservices. Instead a Service Discovery mechanism should be used, such as Eureka or Consul, where services register their API availability with a central server, and clients to request a specific API address from this server.

Diving deeper, there are a number of fundamental flaws or things to consider in this implementation:

Blocking

Due to the synchronous nature of REST, the update stock operation will not return until the notification service has completed its task of notifying all relevant customers. Imagine the effect of this if a particular item is very popular, with 1000s of customers wishing to be notified of the additional stock. Performance could potentially be severely affected and the scalability of the system will be hindered.

Coupling and Single Responsibility

The knowledge that ‘when an item is back in stock, customers should be notified’ is ingrained into the inventory manager service but I would argue that this should not be the case. The single responsibility of this service should be to update system inventory (the inventory aggregate root) and nothing else. In fact, it should not even need to know about the existence of the notification service at all. The two services are tightly coupled in this model.

When Services go Pop

Services WILL fail, and a microservice based system should continue to function as well as possible during these situations. Due to the tightly coupled system described above, there needs to be a failure strategy within the Inventory Manager (for example) to deal with the scenario where the Back in Stock Notifier is not available. Should the inventory update fail? Should the service retry? It is also vital that the request to the Notifier fails as quickly as possible, something that the circuit breaker pattern (e.g. Hystrix) can help with. Even though failure scenarios will have to be handled regardless of the communication method, bundling all this logic into the calling service will add bloat. Coming back to the single-responsibility issue, again it’s my opinion that the Inventory Manager should not be responsible for dealing with the case where the Notifier goes dark.

Pipelines

One way to overcome the coupling of services and to move the responsibility of routing away from a microservice is to follow the pipeline enterprise pattern. Our subsystem would now look like this:

Communication may still be REST based, but is no longer ‘point-to-point’; it is now the responsibility of the pipeline entity to orchestrate the data flows, rather than the services themselves. Whilst this overcomes the coupling issue (and blocking with a bit more work, via asynchronous pipelines), it is considered good practice within the microservices community to strive for services that are as autonomous and coherent as possible. With this approach, the services must rely on a third party entity (the pipeline orchestrator) in order to function as a system and are therefore not particularly self sufficient.

For example, notice that the pipeline will receive a single response from the Back in Stock Notifier (even though there are 2 subscribers), but must be configured in such a way that it can parse the response so that it can subsequently send individual “send email” requests to the Email Notifier for each subscriber. It could be argued that the Email Sender could be modified to batch send emails to many different subscribers via a single request, but if for example, each users name must be included in the email body, then there would have to be some kind of token replace functionality. This introduces additional behavioural coupling, where the Notifier has specific knowledge about the behaviour of the Email Sender.

Asynchronous Messaging

In a messaging based system, both the input and output from services are defined as either commands or events. Each service subscribes to the events that it is interested in consuming, and then receives these events reliably via a mechanism such as a messaging queue/broker, when the events are placed on the queue by other services.

Following this approach, the stock notification subsystem could now be remodelled as follows:

Cohesion is obtained via a shared knowledge of queue names, and a consistent and well known command/event format; an event or command fired by one service should be able to be consumed by the subscriber services. In this architecture, a great deal of flexibility, service isolation and autonomy is achieved.

The Inventory Manager for instance, has a single responsibility, updating the inventory, and is not concerned with any other services that are triggered once it has performed its task. Therefore, additional services can be added that consume Inventory Updated events without having to modify the Inventory Manager Service, or any pipeline orchestrator.

Also, it really doesn’t care (or have any knowledge of) if the Back in Stock Notifier has died a horrible death; the inventory has been updated so it’s a job well done, as far as the Inventory Manager is concerned. This obliviousness of failure from the Inventory Manager service is actually a good thing; we MUST still have a strategy for dealing with the Back in Stock Notifier failure scenario, but as I have stated previously, it could be argued that this is not the responsibility of the Inventory Manager itself.

This ability to deal with change such as adding, removing or modifying services without affecting the operation or code of other services, along with gracefully handling stressors such as service failure, are two of the most important things to consider when designing a microservices based system.

Everything in the world of asynchronous messaging isn’t entirely rosy however, and there are still a few pitfalls to consider:

Design / Implementation/ Configuration Difficulty

The programming model is generally more complex in an asynchronous system compared to a synchronous counterpart, making it more difficult to design and implement. This is because there are a number of additional issues that may have to be overcome, such as message ordering, repeat messages and message idempotency.

Also, the configuration of the message broker will also need some thought. For example, if there are multiple instances of the same service, should a message be delivered to both of the services or just one? There are use cases for both scenarios.

The Very Nature of Asynchronous Messages

The fact that the result of an action is not returned immediately can also increase the complexity of system and user interface design and in some scenarios it does not even make logical sense for a subset of a system to function in an asynchronous manner. Take the Back in Stock Notifier for example, and its relationship with the Subscriber Manager; it is impossible for the notifier to function without information about the subscribers that it should be notifying, and therefore a synchronous REST call makes sense in this case. This differs from the email sending task, as there is no need for emails to be sent immediately.

Visibility of Message Flow

Due to the dispersed and autonomous nature of messaging based microservices, it can be difficult to fully get a clear view of the flow of messages within the system. This can make debugging more difficult, and compared to the pipeline approach, the business logic of the system is harder to manage.

Note: Event based messaging can expanded even further by applying event-sourcing and CQRS patterns, but this is beyond the scope of this article. See the further reading links for more information.

So which communication approach is best when designing your microservices? As with most things in software development (and life??), it depends on the requirements! If a microservice has a real need to respond synchronously, or if it needs to receive a response synchronously itself, then REST may well be the approach that you would want to take. If an enterprise requires the message flow through the system to be easily monitored and audited, or if it’s considered beneficial to be able to modify and view the flow through the system from one centralised place, then consider a pipeline. However, the loosely coupled, highly scalable nature of asynchronous messaging based systems fits well with the overall ethos of microservices. More often than not, despite some significant design and implementation hurdles, an event based messaging approach would be a good choice when deciding upon a default communication mechanism in a microservices based system.

Microservices Like Change

2015-10-16T00:00:00+01:00

Do we always need to be scared of change?

I got kind of excited the other day, I got told the client wanted a change. I know, weird, right? To make matters worse it was just a month before the project was going live. So you would expect the team and I to be perhaps a bit upset and to say, ‘No that change would be too much’ or ‘That means changing the API’. To generally behave like my 6 year old does, when I tell him he can’t have chocolate before bed.

But in fact we were the exact opposite; we wanted the change! Why? Because our design could accommodate it. Our design was paying dividends. We could see this change being implemented with a small amount of work and minimal (or in fact no) effect to the users of our existing API. Actually we could even use this as an opportunity to produce a better and cleaner API for our customers.

Bring it on!

In my previous post Microservices Gotchas I talked about how we were implementing a suite of Microservices; using Java, Apache Camel, Spring Boot, backed by MongoDB. The services were split into varying roles; Adapter services, calling dumb pipelines which called smart data services. If you’ve read or been to talks on Microservices, one of the benefits that always gets mentioned is the ability to accommodate change. Microservices are not all that easy to build from scratch; there is a perceived overhead in the splitting up of the functionality. So the benefit won’t always appear straight away.

In our design we only had to make changes to any affected services, which in our case was only one. We added a new API to an existing service which would consume an already implemented downstream service. And finally the big win was we could move this into production by releasing just one Microservice, with tools like Apollo the change could be made with no downtime in production.

I’m sure not all changes will be like this, but with the right upfront design of your services there is no reason why they can’t be. Using great design tools like the Life Preserver you can model services around components that change together, reducing the impact of those last minute change requests.

Extra Reading

Some additional reading if you get time:

Microservices Like Change was originally published by Capgemini at Capgemini Software Engineering on October 16, 2015.

Microservices Gotchas

2015-04-16T00:00:00+01:00

Don’t get me wrong, I’m not anti Microservice, far from it.

Come and ask me and I’ll give you a lot of reasons why they can benefit any development project. And there are plenty of posts on the web and on this blog that talk about the very good points of implementing a system using Microservices.

However after getting one implementation into production, and now being the majority of the way through a second, I have some observations from the other side of the coin that you may not have thought about.

You’re Airing Your Laundry in Front of Everyone

Yep, there it is everybody, my implementation of your requirements … it’s all there, go ahead take a look. Once upon a time you would implement the “system” and to everyone it would be that box on the diagram that did stuff. Something went in, and things happened and something popped out the other end. That was the “monolith” and in the new world you have multiple services running that make up that “system” with lines of communication draw between them. And that’s great. It means you have a flexible architecture, you can make changes to your services and not affect the whole system.

But it does have a side effect, now everyone sees your implementation and it can turn into something like a mass code review with people you wouldn’t expect pointing things out. You may have noticed I didn’t say “dirty” laundry in the title, as sharing the actual implementation isn’t a bad thing. But managing the message is something you should be ready for. I don’t think I was ready to talk about such low level implementations of our code to everyone.

Think about how you talk and document your system, pitch it at the right level for the audience. Drawing lots of boxes with a spaghetti of overlapping lines can scare the hell out of some people.

That’s Going To Be A Bit Chatty Isn’t It?

Microservices have to talk to each other; That is pretty obvious right? Or is it? When as a developer you’re implementing a single requirement it may not be. Take a step back and look at the bigger picture, and it can become very obvious that lots of small conversations are starting to give you a major headache.

In our implementation we’re communicating with REST over HTTP, which is pretty light-weight, but given high volumes it did cause us some issues. We had to re-think one of our design decisions to resolve it. Luckily we found this out before it went into production and fixed it with very little fuss. How? Performance Testing that’s how.

I can’t stress how important Performance Testing your Microservices is. If you can get an understanding of your volumes and expected performance early on, then you can get on testing it as soon possible. Other “gotchas” to bare in mind:

Make the tests realistic. Is hitting every API at the maximum TPS at the same time the right thing?
Use as real a dataset as possible.
With real volumes.
Push it through as close to a realistic production setup as possible.

This will flush out something that you may not have expected. Usually a piece of hardware or a restriction that’s outside of your domain. Doing Performance Testing early on gives you the time to make the necessary changes in time. And given you’re using Microservices the change is going to be nice and contained, right? Watch this space for a future blog on Performance Testing Microservices.

So How Do I Test These?

You’re writing good code, and you’re writing unit tests. You feel confident. But you still need external testing, the question I got asked a lot by our testing team was how do I test this story, where do I do that? And you can understand why.

We had built a suite of integration Microservices, that you could put into 2 groups; “smart services”, which exposed APIs that do specific jobs and “pipeline services”, which glued together calls to the different smart services (usually in a very specific order) in order to a complete a business task. Despite this, we still found at certain points we got swamped by the amount of test effort required to test every single Microservice. It created a lot of noise within the team.

What helped us was to identify the right Microservices that could be externally tested. Helping to focus the teams effort and reducing the noise of defects that may never occur.

It’s Broken … Somewhere

By their very nature Microservices are a distributed architecture. This brings its challenges when debugging issues; Where do you look? You get told one of your APIs is throwing an error. With the Java “monolith” you had a had a stacktrace that you could work your way down and find the line of code that caused the error. But with Microservices it could be anywhere, especially when your service calls multiple down-stream services, and in some cases in parallel.

So where do you start?

Something simple that developers can overlook is logging and logging the right details, you need to be able to trace a message from it’s entry point to everywhere it gets routed, so having some key data that makes a message unique and “searchable” is a major plus point. Additionally don’t just log on errors, you want to know to where it went and how it got to be there. Monitoring is also a big winner for us; knowing if services are running and performing within norms helps identify issues. We’re using Coda Hale metrics outputting to Graphite and Grafana, but to get a runtime view we’re moving towards implementing Netflix’s Hystrix Dashboard with Turbine in order to see how all our Microservices and their dependencies are behaving.

Extra Reading

Some additional reading if you get time:

Microservices Gotchas was originally published by Capgemini at Capgemini Software Engineering on April 16, 2015.

Why Microservices Are Right For Us

2015-03-06T00:00:00+00:00

I posted previously about the fact we’re “doing” Microservices. At that time I was hedging pretty extensively (as various commenters pointed out). This was predominantly because we we’re pathfinding (or at least broadening the paths for the rest of us that had already been exposed and marked by others just ahead of us).

Well, time has moved on; and while we’re still not fully paid-up cult members, unwilling to drop our skeptical, one-step-at-a-time, always-check-this-is-right-for-us-in-our-specific-situation approach, we are increasingly happy with the path we’ve taken. Why? Let me lay it out for you.

Microservices are Human-Sized

First up, it’s been overwhelmingly noticeable that by adopting Microservices, we’ve allowed ourselves to work on things that are the right size for a single developer to grapple with. By this I mean you can work on them quickly and in isolation, and you can consume them easily too.

What’s more, you can reason about their internals really simply; their cognitive load feels “just right”. Their granularity feels good too - you know when they “do one thing, and do it well” because it’s patently obvious; or more accurately, you know when they’re not, and when its time to split them down. (Others have already written far more eloquently about this than I ever could, but I’d like to add ourselves as a data point which bears this out.)

What is more, not only are they human-sized for us as developers, they are also human-sized for that “auld enemy” of ours - testers. By this I mean that they (Microservices, not testers) can initially be tested quickly, and in isolation - the test work gets a lot closer to our unit tests and, because we’re all REST-based in our APIs, these tests are nice and simple to automate with a myriad of free and well-known and well-CI-CD-chain-integrated tools.

But perhaps even better than both these is the fact that Microservices are also consumer-sized. We write all ours so that with only a Java and Maven dependency installed you can check them out and have them running locally with two commands:

git clone
mvn spring-boot:run

We set up our code so that, when the Microservice starts up, it’s automatically in stub-mode, by which I mean it’s downstream Microservice and other dependencies are stubbed out. The range of requests you can send in, and the various valid and failure states you’ll mimic as a result are specified in standard README.md files at the top of each Microservice git repo.

This, plus the bundled and served Swagger docs, Hystrix Dashboard and Spring Boot / Coda Hhale Metrics monitoring JSON feeds, means everyone is a lot more self-supporting and doesn’t need to depend upon shared server environments with the associated time-slicing and wait for the next release.

Microservices are Laptop-Sized

Picking up right where our last point left off is our next benefit; adopting Microservices has allowed us to throw out a lot of the heavyweight-and-rarely-used shared development infrastructure, allowing us to get a lot more as individual developers out of the grunt of our development and test machines. Running Tomcat (we’re not on Jetty yet, but we will be eventually - its inevitable) and possibly an in-memory-db (or even a local Redis or MongoDB) is a LOT lighter on resources and quicker to turnaround after a change than a traditional Java EE app server. Despite this, we’ve lost none of the monitoring or configurability we’ve come to expect from the more “full-featured” EE cousins; none that we’re missing anyway (see above). And if we do need to run up more than one Microservice locally? Nae bother.

And before we move on, it’s worth us pointing out that this is all without the much-discussed Docker currently being in the mix. (We’re looking at moving to using this in Dev but its not imminent, though our minimal platform-dependencies should make it very simple once we find the time to do it properly.)

Microservices are Team-Sized

Still on the sizing, here’s our penultimate benefit of this post: Teams of 3-5 folks typically work on a Microservice together. This is a good fit for us - we typically Pair Program, or review all of each others code, which keeps it clean, focussed and legible. We’ve also found that a right-sized Microservice is a great size for a productive team-discussion, with collectively-owned outcomes. When there is disagreement - and there is occasionally - then quick spikes of a few days max produce great results.

One thing we’ve not tried, but which appeals to us, would be to take this approach one step further and let developers outside of the service-team make changes to a Microservice’s code via an OSS-like pull request model. Of course this is predicated on each service having its own git repository - but we have that - and a PR / code review model approach to working - but we have that too.

Microservices are Truly “Agile”

Last benefit time. For this one, let’s push beyond the “size” theme. I know I’m walking into a minefield here (hence the inverted commas) but I mean Microservices are “agile” in the true sense in that they:

keep the humans at the centre (see everything in this post up to now)
help keep the focus on running, tested (TDD’d?) code
absorb change well (multiple parallel versions and refactoring!)

The first bullet we’ve already done to death. The second we’ve mentioned and I won’t go into any more. The third however, bears a little exposition.

From the start we knew that we’d be in the situation at some point where we’d have to run multiple versions of a Microservice. Fine, we versioned the URLs (major number only if you’re wondering). We soon afterwards extended this so that multiple instances of a given service (even the same version) could be run alongside each other. To do this we had to ensure they didn’t use each others port(s), config, output directories, and we had to add an instanceId to all the metrics we shipped else they looked like one big instance in things like Graphite / Grafana. With this in place, and a backwards-compatibility policy set out we were good to grow. But that’s growth in the large, external to each Microservice. What about change internally? It turns out, that’s pretty great too.

Because they are by their very nature self contained and “simple”, and exist behind cleanly defined interfaces we’ve found our Microservices a lot easier to refactor. Changes are smaller as the codebase is smaller. The changes are easier to grok for a similar reason. And because of this, most forms of debt accrete a lot more slowly. Beyond this, because you’re in a completely separate codebase, looked after by a smaller team, experiments are easier to undertake, and are hidden from everyone else. That helps free folks from the tyranny of “that’s not how we do it”, and lets them make decisions based on their requirements, and their requirements alone. Now, in my opinion, that’s agile.

Conclusions?

That’s it for this post - we have other potential benefits that we’re tracking as I write, and rest-assured, I’ll post them as and when we’re sure they really are “beneficial”.

In the meantime, if you have any comments / questions, please share them in the comments section below, or tweet we on @al94781. I’d love to hear from you.

Want to Work With Us?

Fancy working as part of the team I’m talking about, or on any other number of exciting Digital projects? Let us know because, we’re always hiring.

Why Microservices Are Right For Us was originally published by Capgemini at Capgemini Software Engineering on March 06, 2015.

Solving problems being technology agnostic

2014-12-01T00:00:00+00:00

We are used to building systems and solving problems under high quality requirements as developers.

When you are creating a whole system with its own behaviour and idiosyncrasy your solution is probably going to have some specific characteristics due to your requirements, your particular priorities and the context that you build it to.

There are many factors that matter when creating new architecture. Maybe the scalability of your system is a key factor for you. Maybe the requirements of your system can change frequently, so flexibility would be crucial. Maybe you have legacy code dependencies. Maybe you need to give priority to readability over performance. Depending on the capacity that your system needs to support, you could make different decisions. Valid approaches for monolith systems could not apply for decentralised or distributed systems.

Talking now about a lower level of abstraction, you will try to make the most of the language, technology stack and programming paradigm (imperative, functional, OOP, etc) that you choose or that you have to use. So this will yield some peculiarities to your creation.

All this being said I find that we can identify some principles and techniques applicable for shaping almost any solution so sometimes we should think in a more “technology agnostic” way to make the right moves. Our system could be a “Drupal module”, an “Automated Deployment Pipeline” or an “Enterprise Web Application”. I think there are principles generic enough for applying to such different systems and that can help us to ensure quality and good results regardless of the nature of the problem when building a solution:

Choose the most appropriate form of separation for your problem and divide the system into reusable pieces in order to reduce the coupling between parts with different purposes. A piece is something that is able to interact with other components using a given protocol or “idiom” but that can work by itself (e.g. you will usually want to decouple your business logic from the framework, the back-end from the front-end, the database technology from the application, you could want to decouple the physical deployment in the form of a n-tier architecture). SoC.
Fewer components are better. Humans are not good at understanding complexity.
Create abstractions of complexity. Abstraction Principle.
Keep it simple. KISS.
Isolate details.
Define a clear boundary for responsibilities. One thing for doing just one thing. Single responsibility principle.
More frequent small changes are better than big occasional changes.
Don´t duplicate things. Reuse everything that can be reused. DRY.
Use meaningful names but without involved and unnecessary commitments.
Choose conventions. Be consistent. Use the same approach for solving the same small problem several times.
Don´t procrastinate. Don´t increase debt. Do it now.
“Yes” is forever. If you’re not sure about something new, say no. You can change your mind later, otherwise you will probably be technically committed forever.
Automate everything that can be automated.

So having these principles in mind can be a helpful reference for driving your decisions when solving problems being technology agnostic. Here is a list with some resources that drove me to embrace these principles:

Solving problems being technology agnostic was originally published by Capgemini at Capgemini Software Engineering on December 01, 2014.

Microservices - A Reality Check(point)

2014-10-17T00:00:00+01:00

It’s reached the point where it’s even a cliche to state “there’s a lot written about Microservices these days.” But despite this, here’s another post on the topic. Why does the internet need another? Please bear with me…

We’re doing Microservices. We’re doing it based on a mash-up of some “Netflix Cloud” (as it seems to becoming known - we just call it “Archaius / Hystrix”), a gloop of Coda Hale Metrics, a splash of Spring Boot, and a lot of Camel, gluing everything together. We’ve even found time to make a bit of open source ourselves - archaius-spring-adapter - and also contribute some stuff back.

Lets be clear; when I say we’re “doing Microservices”, I mean we’ve got some running; today; under load; in our Production environment. And they’re running nicely. We’ve also got a lot more coming down the dev-pipe.

All the time we’ve been crafting these we’ve been doing our homework. We’ve followed the great debate, some contributions of which came from within Capgemini itself, and other less-high-profile contributions from our very own manager. It’s been clear for a while that, while there is a lot of heat and light generated in this debate, there is also a lot of valid inputs that we should be bearing in mind.

Despite this, the Microservices architectural style is still definitely in the honeymoon period, which translates personally into the following: whenever I see a new post on the topic from a Developer I respect my heart sinks a little as I open it and read… Have they discovered the fatal flaw in all of this that everyone else has so far missed? Have they put their finger on the unique aspect that mean 99% of us will never realise the benefits of this new approach and that we’re all off on a wild goose chase? Have they proven that Netflix really are unicorns and that the rest of us are just dreaming?

Despite all this we’re persisting. Despite always questioning every decision we make in this area far more than we normally would, Microservices still feel right to us for a whole host of reasons. In the rest of this post I hope I’ll be able to point out some of the subtleties which might have eluded you as you’ve researched and fiddled, and also, I’ve aimed to highlight some of the old “givens” which might not be “givens” any more.

The Good

They are the right size. And by “right” I mean “right” for a developer, for source-control, for CI, for documentation, for release/upgrade, for scaling, for resilience, for APIs/consumption/composition into things-larger, and finally for replacement. For all these things they’re all good
They get things out of the way. With Microservices we’re coding in Java again, well, in Camel-Java-DSL, and this lets us think like software engineers, rather than JEE architects or Spring-Bean-experts. It means we can TDD, and TDD like we meant it, and that means we can refactor, and keep our code and designs looking like we care about them. (Better still we haven’t had any weird classpath error issues to debug from our JEE server as we don’t have them. And when we’ve had problems on the wire, because we’re using HTTPClient direct, we can get into it and find out whats going on far more quickly)
They’re more predictable. Perhaps the biggest gain is because they’re “micro” they’re easy to comprehend. Now I’m not forgetting the dangers of combinatorial complexity (we’ll get to that later) but because we’re working with small, well-tested cohesive components here, and stateless, idempotent, circuit-broken ones at that, things are a lot more likely to do what we think they will do. The spare-cognitive-load gains from that as someone ultimately responsible for all this is immense
They’ve made us question how we do things. Accepted wisdom isn’t accepted any more. The change in approach has made us question far more than we would on a “regular” project. Because this fundamental part of our job has changed, what else might have changed too? The resulting flowering of creativity in the team has been exceptional, and its been exciting to see it unfold. What’s more, I’ve seen properly reusable code coming out of the teams for the first time in my career. It’s almost as if all this component-thinking at the microservice level is infecting everything else. #winning
It’s fun, it’s exciting, and actively doing things that are new (and this does feel new) keeps you on your toes far more than a “standard” (read: JavaEE) approach would. That’s a good thing. A great thing

The Un-Good

And yet its not all #winning. Perhaps it’s the lack of balanced opinion in the general chatter that makes me feel the fear I mentioned earlier. Most posts I read are just so sycophantic on the topic, and the world really doesn’t need another one of them. So, deep breaths, lets dig into that a bit more and present some of the reasons Microservices might not be for you.

First up, some honesty. We’re finding that despite all the noise in this space very few folks out there are actually doing this, and even fewer doing it in a public manner. We however are. Finding that very few others are with you can be a little scary at times. It means you actually have to do research and make your own decisions which for most of us in the safe world of Java Enterprise Development is a new experience. Consequently, we’ve staked our professional reputations on this, and we’ve got a lot less to hide behind - we’re on the front line of all this challenging of accepted wisdom
Things won’t “just work” when you glue them together; and things which do work might not have the greatest documentation in the world. Many many decks on slideshare refer to all the bits you’ll need to get up and running in seconds - but there is a impedance mismatch between many of them, and as an early adopter you need to fix that. Having said this, the other thing all these projects have in common is a vibrant community. In most things in this area the code is under active development, and so is the documentation. If you want to get involved and help, folks are very pleased to have you along for the ride. That’s great, but it does slow things down a bit. It also means you might be ahead of the curve on the major libraries you rely heavily on - for example we plugged Hystrix into Camel for our own Circuit-Breaking purposes before they added a CB of their own in the latest (2.14.0) version. We also chose CXF for our REST APIs (when the rest of the world was going Jersey) because it was a first-class citizen in Camel, only to find the exposure of Camel Routes as REST wasn’t at that point incredibly mature. (Note: It now is, it’s even got Swagger support)
Related to the point above, you need to remember you might make wrong bets. I already mentioned that we went CXF when the rest of the world seemed to be going Jersey. To be honest that’s not hurt us too much - we have what we need. We’ve also been right in adopting Hystrix, Archaius, and looking at Eureka, Ribbon and Zuul as they have only just been announced as being supported by the new “Spring Cloud”. It might however have been a mistake to go for Spring Boot - Fabric8 is getting more and more mature by the day, and solves a lot of problems we might face at some point in the future, or have had to code around ourselves (i.e. by building our continuous deployment pipeline with Jenkins, Puppet and Capistrano)
You end up with a lot of moving parts. You’re confronting the Fallacies of Distributed Computing head on, and tackling them each in turn, in everything you build. You end up leaning more heavily on tools like Maven (in our case) or Gradle (we’re evaluating) , and thinking about versioning, and running concurrent versions from the beginning is key. You also end up needing to be able to “boot-up” a component with a load of stubs for all their dependencies so that you can run individual bits on their own
This is a new way of thinking. This is INTEGRATION at EVERY LEVEL. In the JavaEE world we never thought of threads because we weren’t allowed to. We’ve found that models like Scala’s Akka are a great mental tool for thinking about these problems, even if we’re not using the frameworks, but we had to get there the hard way
Following on from the two points above you end up having to make some concessions in order to be able to cope with all this - the biggest one is that you need to embrace immutability (in code and deployables, and environments) and discard state. This makes a lot of things easier, and, if you invest the time in proper Continuous Deployment and elastic scalability enablement
You end up a lot closer to the metal, and the network. We’re working with HTTPClient directly, and are looking at Protobuf / Thrift / Avro for inter-machine comms (Camel lets us do inter-process, intra-machine comms quite nicely). Latency also hits you front and centre. Again, having to deal with this head on is no bad thing, but it’s not the usual state of being for a traditional Java developer.
Inter-team comms - because you easily gain the benefits of small teams (2-3 dev) working on individual “services” hidden behind a clean API you end up hitting the many patterns finely articulated by Eric Evans in Chapter 14 and beyond of his Domain Driven Design: Tackling the Complexity in the Heart of Software (perhaps the best part of that fine volume). If you know to expect them then this isn’t too bad, but it will happen. This has meant that we’ve been looking at Swagger as a nice way of documenting all our APIs, internal and external, to reduce the interruptions when team A needs to consume component from team B
You end up with variation. Yes, we’ve componentised, have created some common components, and had various pull-request submissions back to the community accepted, but you still end up with various ways of doing the same thing. Now this isn’t always a bad thing, but it means that you need to rely much more on keeping a clean code(base). We review every pull request, with approval being required from >= 2 developers. Typically we aim for these approvers to be from outside the team. We’ve also instituted an internal RFC mechanism (stolen from Carl Quinn who I believe has used it at both Netflix and Riot Games to manage changes required by the dev teams he leads)
You end up flooded by data. Everyone says that you need a lot of monitoring. They’re right. Its easy to add too. So easy in fact that we ended up submitting a pull request to Hystrix which allows you to filter what is produced because we were flooding the UDP port of our monitoring servers. We’re lucky in that we have a great DevOps team who put all the supporting infrastructure in place for us to take advantage of all this too. But we’ve needed to become expert in setting up just-the-right-amount Grafana dashboards. Its another skill to learn.
Dev becomes support and Support becomes dev. this one relates back to Number 5. Folks who are in Support because they want to support things that look like other things will get a shock. We’ve been lucky - our support guys have been very keen to learn something new. We’re getting them to work on new features with us as a means of teaching them how things work. Additionally, because things are so new, we have to get a lot more involved in support. So much so that we’re trying to get a big TV for our dev area to put our Hystrix / Grafana Dashboards on permanently so we know how things are looking

And that’s it. I’d like to point out again that so far, we’ve been very pleased with our decision to adopt this architectural approach. But we’re still keeping our eyes open. Remember, there’s No Silver Bullet.

A Reading List

Before we close, here’s a reading list of the things we’ve found most useful in our journey to here. Please add a comment if you have any other suggestions of items to add:

Microservices - A Reality Check(point) was originally published by Capgemini at Capgemini Software Engineering on October 17, 2014.

Trade-Offs

2014-10-16T00:00:00+01:00

Why Stuff Goes Slow

waiting
blocked
taking the long way round
forgetting what you knew

Why Stuff Gets Complicated

Engineering around why stuff goes slow

Trade-Offs was originally published by Capgemini at Capgemini Software Engineering on October 16, 2014.

Capgemini Software Engineering

Devoxx 2024

AI Again

AI and Vectors

Thoughtful Architecture

More microservices

Fear of Rust

Crashing and burning

Continuous Change

Micro Frontends: an Introduction

Microservices & Distributed Systems in General

User & Developer Experiences

Micro Frontends: a New Alternative

Integration

State & Communication

Potential Problems

Conclusion

Enterprise Architecture DocOps

Introduction

docToolchain

DocOps with Azure DevOps

Conclusion

Appendix: PlantUML diagrams source code

Sequence type

Class type

JSON type

DocOps workflow

The Conservation of Complexity in Software

Complexity in Architecture

Complexity in People

Complexity in Phasing

In Conclusion

Combining OAuth and JWT to gain performance improvements

Techniques to improve Application Design

Start small

Keep it simple

A practical design not an ideal one

Feedback

Foresee

The Unbreakable Design - There is no such thing

Is REST Best in a Microservices Architecture?

REST

Blocking

Coupling and Single Responsibility

When Services go Pop

Pipelines

Asynchronous Messaging

Design / Implementation/ Configuration Difficulty

The Very Nature of Asynchronous Messages

Visibility of Message Flow

Further Reading

Microservices Like Change

Do we always need to be scared of change?

Bring it on!

Extra Reading

Microservices Gotchas

You’re Airing Your Laundry in Front of Everyone

That’s Going To Be A Bit Chatty Isn’t It?

So How Do I Test These?

It’s Broken … Somewhere

So where do you start?

Extra Reading

Why Microservices Are Right For Us

Microservices are Human-Sized

Microservices are Laptop-Sized

Microservices are Team-Sized

Microservices are Truly “Agile”

Conclusions?

Want to Work With Us?

Solving problems being technology agnostic

Microservices - A Reality Check(point)

The Good

The Un-Good

A Reading List

Trade-Offs

Why Stuff Goes Slow

Why Stuff Gets Complicated