Capgemini Software Engineering

The ramblings of an open source newbie

2016-05-13T00:00:00+01:00

As with most developers, I’ve used open source software. As with most developers I expect to find this free, well documented, easy to use software within a few clicks and I expect it to perform and to deliver the advertised functionality.

Having never contributed back to any open source projects, I was lured into helping with the newly created spring-boot-capgemini project that would provide some useful common spring boot starters. With Spring (and I still don’t know how this happened) being very new to me, it seemed a useful and effective way for me to put some of the theory into practice.

I’m no writer, so some bullet points with some of my thoughts, experiences and perhaps some prompts for discussion will have to suffice.

“I think I’m alone now”

Joining the project opened up multiple lines of communication via github issue tracking, emails, & slack.

I missed the face to face contact and the quick “can you just look at this over my shoulder, it’s bound to be something stupidly obvious”. Slack was bamboozling me with too much detail - commits, builds and IMs notification were popping up continuously.

I suggested a few changes to minimise the noise, but this resulted in a longer than expected discussion and debate, which again I felt could have just been sorted quickly had we sat down and discussed.

I struggled when I got stuck with things - this wasn’t like my day job, I couldn’t tap up the many experts available to me. The usual suspects (Google, StackOverflow) were useful to a point, but i did spend some time staring at slack, waiting for one of the other guys to come on line!

Learning curves

As previously mentioned, being new to Spring and Spring Boot I expected to struggle - but there were far more hurdles in the way than I expected.

Gradle? Where’s Maven? GitHub? Where’s SVN? Jenkins….where are you?

I see this as a plus - being dragged out of your comfort zone and learning new technologies because you need to. You have some context and specific goals, so you are more driven and focused to get past specific and relevant hurdles, rather than picking up something new, getting “Hello World!” out of the way and then wondering where to go now.

“What’s the number for the guys in support?”

I took on the task of integrating the repo with a build system. I realised how spoiled I was on my day to day project work, with Jenkins a click away and someone else to fix and maintain it.

Finding a solution compatible with our private repo was a challenge, but circleci.com was chosen, and again, configuring it, picking up YAML and getting it all working seamlessly, was down to me.

On the flip side, you’re not bogged down with someone else’s choices - your project can have what you like as long as you sort it out and maintain it, although I imagine on an open source project of any serious scale, democracy and/or mob rule and even a dictatorship decides the technology and infrastructure.

Quality Control

Checkstyle issues breaking the build! I wasn’t expecting that, but at its inception it was the right time to make decisions like this.

No time or cost considerations, no imminent deadlines - why not strive for the highest quality software? I have heard horror stories of simple pull requests turning into scathing attacks on people’s work, warring factions and even submitters not toeing the line being banned!

This story, seemingly one of many examples of the Linux guru Linus Torvalds throwing his toys out of the pram, are testament to the “keyboard warrior” culture present in some open source environments.

Every line of submitted code being reviewed, every checkin requiring a test - this is how it should be done. Does this discourage contributors, or does it just weed out the bad/lazy ones?

Documentation is King

“The code is the best documentation” - a quote I’ve heard several times, but in open source, surely the README.md file is the most important file in the repo? This is the project’s marketing, sales, and support, all in a file which will get rapidly scanned and form the basis of an opinion or decision - is this worth looking at let alone forking? Is it a serious contender compared to the competition? Is this project still even active and how well will it be supported if I choose to use it?

Longevity

The spring-boot-capgemini repo started off strong, enthusiasm was high and activity was sustained and productive. It tailed off however, day to day account work is always higher priority, and I wonder perhaps this was inevitable?

When someone is paying for software, expecting features on a certain date, there is a drive to deliver, deadlines to hit, functionality to provide. On larger, more established open source projects I imagine this is the same, but how does a small open source project get to that point?

Conclusion

In conclusion, I think I have ended up with more questions than answers.

The more I write the more I realise that the spring-boot-capgemini project is too small, too early in its lifecycle on which to base anything than a first impression of open source. I don’t feel I can make any firm opinions or judgements until I have looked into and/or contributed to a number of open source products, and that goes beyond just pulling in the jar as a dependency.

That’s the great bit - the transparency, clue’s in the name, “open” source ! I can find how the libraries I use day to day are written, tested, documented, versioned, and released. I don’t need to be an active contributor, until I feel ready, and I choose what I contribute.

I’m sure in time I’ll look back on this as a bit of a naive blog entry, and hopefully come back with some answers, backed up with experience.

The ramblings of an open source newbie was originally published by Capgemini at Capgemini Software Engineering on May 13, 2016.

Continuously deploying Apollo with Wercker

2015-07-03T00:00:00+01:00

When deploying Apollo it will create the cluster infrastructure in the cloud using an image built by Atlas and Packer. Additionally, it will provision and configure the new machines providing a full PAAS ready for deploying containers across the datacenter.

When dealing with several layers of abstraction and using many different technologies (Bash, Packer, Terraform, Ansible, etc.) it is fairly easy to break things during development.

We need to make sure we are not committing syntax errors, that every commit is deployable and that the behaviour of the platform with all the bits communicating to each other works as expected.

Changes need to be continuously integrated and driven by tests. For this purpose we chose Wercker out of a list of available CI as a service tools like Travis, circleci, Drone and Shippable

“Wercker is an open automation platform for developing, building and delivering your applications” from the simplicity of a YAML file and it is highly customizable via custom steps

For achieving our continuous deployment approach we’ll use two different pipelines: build and deploy

Build

“Builds are the result of a run-through of the steps in the wercker pipeline. If all steps in the pipeline are successful, a build has passed”.

Every time a Pull Request is created the build steps below will run:

Syntax checks against all our json packer templates using packer validate.

script:
  cwd: packer/
  name: validate packer template for AMI
  code: ./packer validate ubuntu-14.04_amd64-amis.json

Linting checks against the ansible code using our custom step-ansible-lint that relies on https://github.com/willthames/ansible-lint.

capgemini/ansible-lint:
  name: run ansible-lint against the site.yml playbook
  playbook: site.yml

Very simplistic Bash unit testing for ensuring our bash functions do what they are meant to do.

script:
  name: scripts testing
  cwd: bootstrap/tests
  code: |
    /bin/bash test.sh

In the future we might be adding a new step here for “packer push” the images into Atlas when merging into master branch.

You can see the feedback for every build in wercker:

Deploy

Wercker allows you to deploy your application into supported platforms like Heroku or OpenShift or you can create you own Deployment Target.

Every successful build from any specific commit can be deployed either automatically or manually.

At the moment we are continuously deploying Apollo into Amazon and Digitalocean using a step-apollo-deploy.

deploy:
    steps:
    - install-packages:
        packages: wget unzip
    - pip-install:
        requirements_file: "requirements.txt"
    - capgemini/apollo-deploy:
        cloud: $CLOUD
        artifact_name: $ARTIFACT_NAME
        artifact_version: $ARTIFACT_VERSION
        terraform_version: 0.5.3
        run_tests: true

Environment variables can be set for every target you create in wercker:

We have created step-apollo-deploy step for:

Installing a given version of Terraform.
Deploying Apollo in parallel into multiple clouds.
Running different types of testing inside the vms. At the moment Serverspecs and docker bench security.
Destroying the whole platform.

Every change in step-apollo-deploy is also continuously integrated by wercker itself using step-validate-wercker-step:

box: wercker/default
build:
  steps:
    - validate-wercker-step

So in essence with both “build” and “deploy” in place we have the ability to automatically triggering a fully tested ephemeral Apollo deployment for any specific commit.

Test

We use ansible roles for installing and running dockerbench and serverspecs covering components of Apollo:

Mesos
Marathon
Docker
Weave
Consul
Registrator
Dnsmasq
HAProxy
Zookeeper

Testing during a deploy can be optionally enabled using a toggle environment variable.

export APOLLO_serverspec_run_tests=true
export APOLLO_dockerbench_run_test=true

Dockerbench

We have recently decided to try and integrate dockerbench into our test suite.

The Docker Bench for Security is a script that checks for all the automatable tests included in the CIS Docker 1.6 Benchmark . It gives INFO, PASS and WARNING messages for each of the tests that run. We run tests at deploy time, to check we haven’t slipped any unwanted security risks into our build. Currently we only test if WARNING’s are above a certain configurable threshold.

The dockerbench project is still fairly young, and is still to settle on a standard way all tests can be run across any environment, whilst having configurable rules. We think this is a great project, as it allows us to bring security into our Continuous Integration cycle, which can decrease attack vectors and catch risks earlier in development cycles.

Serverspecs

Apollo default deployment provides 3 machines playing mesos master role and running zookeeper and marathon. As some configuration like the ips can be assigned on deployment time we need to populate some of the serverspecs dynamically by using an ansible j2 template.

describe command("curl --silent --show-error --fail --dump-header /dev/stderr --retry 2 http://{{ marathon_hostname }}:8080/v2/info") do
  its(:stdout) { should match '.*"master":"zk://"{{ marathon_peers|join(',') }}/mesos".*' }
  its(:stdout) { should match '.*"zk":"zk://{{ marathon_peers|join(',') }}/marathon".*' }
  its(:stdout) { should match '.*"name":"marathon".*' }
  its(:stdout) { should match '.*"mesos_user":"root".*' }
  its(:stdout) { should match '.*"executor":"//cmd".*' }
  its(:exit_status) { should eq 0 }
end

Below are some examples with more of the serverspecs we run for marathon:

describe docker_container('marathon') do
  it { should be_running }
  it { should have_volume('/store','/etc/marathon/store') }
end

describe port(8080) do
  it { should be_listening }
end

describe command("/usr/local/bin/marathon-wait-for-listen.sh") do
  its(:exit_status) { should eq 0 }
end

describe service('marathon') do
  it { should be_running.under('upstart') }
end

describe command("curl -s -XPOST 127.0.0.1:8080/v2/apps -d@spec/marathon/sample.json -H \"Content-Type: application/json\" && sleep 3") do
  its(:exit_status) { should eq 0 }
end

describe command("curl -s 127.0.0.1:8080/v2/deployments") do
  its(:stdout) { should match '[]' }
  its(:exit_status) { should eq 0 }
end

describe command("curl -s 127.0.0.1:8080/v2/apps/serverspecs-app") do
  its(:stdout) { should match '.*"id":"/serverspecs-app".*' }
  its(:stdout) { should match '.*"tasksRunning":1.*' }
  its(:exit_status) { should eq 0 }
end

Feedback

Finally we get the feeback for the deployment in every cloud in our Slack channel using wantedly/pretty-slack-notify:

after-steps:
    - wantedly/pretty-slack-notify:
        webhook_url: $SLACK_WEBHOOK_URL
        channel: apollo
        username: wercker-bot

You can check out the full wercker.yml file here and below is a video showing the whole cycle:

What’s next

If for your use case you need to keep track of the state of Terraform you might find this Atlas and GitHub integration useful.

Check out our upcoming posts if you want to hear more about CI, CD and our Continuous Delivery approach for services running inside docker containers deployed by Apollo, running Serverspecs into ephemeral containers via “docker exec” across multiple environments.

Continuously deploying Apollo with Wercker was originally published by Capgemini at Capgemini Software Engineering on July 03, 2015.

Automation as a way of thinking... and docker

2014-11-10T00:00:00+00:00

Automation is a key and essential fact when solving problems and assembling pieces together. By automating you only need to do the same thing once, reducing the possibility of human errors so it helps to increase quality, efficiency and productivity with less effort.

The everyday life

It doesn’t matter what programming language, technologies or tools you use. It doesn’t matter what sector of Software Engineering you are focused on and it doesn’t really matter which discipline you work on. I would say automation can be more of a way of thinking that could even be applied to almost everything in your everyday life.

When you automate a process you are creating and increasing productivity whether the process is “a virtual machine provision”, “the steps you follow since you wake up until you get into the office” or “flirting with somebody”. You are giving an automatic solution to a problem by creating a system with the ability for saving, reading and reproducing a series of steps for achieving an objective that is going to save you time in the future.

When you face and solve the same or similar problems again and again you soon realise patterns and known techniques (e.g: DRY, Information hiding, Open/closed principle, etc.) that will help you create a better design. Once you identify a regular pattern for solving a problem, you can then reuse this pattern every time you are building an automatic solution with common characteristics. But these patterns are not magic, they are always dependent to the context so you will need to suit them for fitting in your context. (E.g: There is no universal way to “break the ice with” a person as the reaction can be different depending on who you are approaching…)

Going back to the IT world

Going back to the IT world, we are always trying to find, create and contribute to the best tools for automating our processes and ensuring high quality, reliable and scalable solution delivery to any users and clients we work with. In this repo you can see the technical details of one solution that fits together a traditional Virtual Machine, puppet, docker, fig and a behaviour-driven development framework (E.g. Behat) for automating the creation and execution of an ephemeral User Acceptance Testing environment.

We use the power of the dockerfiles for reusing the “uat_environment” puppet module (a wrapper over selenium module and its dependencies) and create a docker image with selenium running.

A “behat service” will be created to check if the selenium container is ready using the script inspired by docker-wait so it will run behat, which code is passed as a volume to the container.

We use fig for running the containers and managing our basic UAT environment. You can add as many selenium services as you need to your fig.yml and run different behaviour-tests projects simultaneously because as we are linking containers we don’t have to worry about port collision troubles.

selenium:
 build: dockerfiles/selenium-image
behat:
 build: dockerfiles/behat-image
 volumes:
  - behat/:/behat/
 links:
  - selenium:selenium

Run “fig up” or any other fig command for automating and orchestrating your services.

Here is a list to summarise some of the benefits:

As we only need to run processes in isolation here docker containers will be fast and less consuming than traditional Virtual Machines.
By using “fig” you can easily scale your UAT environment.
As we are passing the BDD Framework as a volume, the environment is totally decoupled from the code.
The environment can be recreated and destroyed as many times as you need without being consuming resources unnecessarily in the meantime.
Everything is under version control system and is reusable.
You can now run “fig” from Jenkins or any other Continuous Integration tool.

Automation as a way of thinking... and docker was originally published by Capgemini at Capgemini Software Engineering on November 10, 2014.

Automated testing for POODLE

2014-10-10T00:00:00+01:00

Why should systems and infrastructure not be treated in the same way as other software components, especially when it comes to implementing security concerns. With today’s POODLE announcement of another SSL vulnerability it makes sense to add infrastructure tests to your regression tests

ServerSpec is a solid way of testing for this, and can be done as follows:

describe command 'openssl s_client -connect localhost:443 -ssl3 < /dev/null' do
    its(:exit_status) { should eq 1 }
    its(:stdout) { should match /no peer certificate available/ }
end

Use of the < /dev/null is to force the openssl client to terminate instead of waiting for input from the shell as we are only interested in the key exchange.

Automated testing for POODLE was originally published by Capgemini at Capgemini Software Engineering on October 10, 2014.

Reflections on Symfony Live London 2014

2014-10-10T00:00:00+01:00

At the end of September, I went to my first “non-Drupal” PHP event, Symfony Live London 2014. With Symfony components becoming a large part of Drupal 8 it was an excellent opportunity to learn a bit about what it all means, and meet the Symfony community. I’ve dabbled with Drupal 8, and we use some Symfony components in our existing Drupal 6 and 7 projects, so I wasn’t coming in completely cold.

Decoupling for re-use

Throughout the conference, I was struck by how much focus there was on re-use, decoupling and framework interoperability. It was the most common thread running through both days - from a workshop diving into the depths of the HTTPKernel, talks on avoiding The Dependency Trap and building composable HTTP middlewares using Stack - as well as one I couldn’t make it to, which covered Decoupling with design patterns - all driving home a message of using frameworks to give you power, but to ensure your code has minimal dependencies and maximum opportunity for re-use.

“Decoupled code is easier to maintain, easier to re-use, easier to read”

Jakub Zalas

The Naked Bundle

Top among these presentations for me though, was The Naked Bundle presented by Matthias Noback. Matthias focused on how the bundle (Symfony’s equivalent of a module) could be stripped back to the minimum Symfony specific code, and merely act as a bridge between the framework and your business logic (which, he asserted, has no place belonging in a framework specific implementation, and should be in its own standalone, interoperable library).

“The framework is for you, but your code doesn’t need it”

Matthias Noback

The short summary of The Naked Bundle was the premise that it should be easy to take your business logic and expose it to a Symfony bundle, a Laravel package, a Drupal module - it’s all PHP after all. Rather than repeat the talk here, I’ll direct you to the excellent slides describing a number of approaches to meet this goal.

Drupal modules

The concept of a Naked Bundle has some direct relevancy for the Drupal world. We’re in the middle of the Drupal 7 lifecycle, many of us still supporting or evolving Drupal 6 sites, and most of us looking forward to Drupal 8 and the changes that it will bring to the Drupal ecosystem. It’s therefore not unreasonable that a large number of Drupal developers are building functionality which may be in use on three versions of Drupal, or maybe more, not to mention the likelihood of Drupal developers looking at other solutions for some sites, whether it be Symfony, Laravel or something different.

The Naked Module?

So then, is the Naked Module a concept worth considering? Why not move as much business logic as possible into interoperable PHP libraries which get pulled into Drupal via a bridge module? There’s obviously a fine line to tread in ensuring you don’t throw out all the benefits of Drupal’s core and contrib functionality, but if building something specific to your domain then there’s a lot to be said for this approach:

Portability - PHP libraries are more easily moved between Drupal versions, or to other frameworks altogether. Migrating functionality from one version of Drupal to another therefore becomes much less of an issue, as your custom code is now less coupled to specific versions.
Testability - a standalone PHP library is more easily unit testable, regardless of whether it’s going to be used in Drupal or not.

How?

In projects at Capgemini, we’re already using several approaches which lets us get some way towards this concept:

Writing domain specific logic in standalone libraries, for example, code to handle creation, validation and manipulation of business objects implemented in PHP classes, and called out to from Drupal hooks
Integrating with internal or external remote services via a service wrapper which can again be called from within the Drupal module
Using the Composer Manager to include these PHP libraries in our Drupal installations so that they can be managed separately to site or module repositories
Using a shared ORM and DBAL such as Doctrine to access custom database objects consistently regardless of Drupal version or framework, particularly when they don’t need to be Entities

Most, if not all, of these approaches are already in use in the community, for example Commerce Guys recently released a generic PHP addressing library to handle creating, manipulating and formatting postal addresses for shipping or billing across different countries. Providing this as a generic library rather than hiding it in a Drupal module means the support for this logic can benefit the entire PHP community.

Libraries first

As Matthias stated in his Naked Bundle talk, it’s only sensible to seek practical reusability. In the case of Symfony bundles an example of an allowed dependency would be the HTTPFoundation classes and trusting the HTTPKernel.

In Drupal, we’d want to take advantage of existing hooks, and well defined, well tested APIs rather than re-inventing things. However, with a libraries first approach, modules can be made smaller, slimmer, avoid framework specific conventions and dependencies, and simply expose resources to add rich domain specific functionality for easier porting to other versions or frameworks.

Reflections on Symfony Live London 2014 was originally published by Capgemini at Capgemini Software Engineering on October 10, 2014.