Tom Driven Dev

Going Beyond Test Coverage

2021-03-11T14:31:32+00:00

Why are you writing automated tests for your project? I hope the answer is to find defects! Writing tests is not free - someone somewhere must design, develop and maintain them. All of this is expensive so we must get something out of our investment.

So how are you proving that your automated tests are valuable and what is the value they deliver?

What is Value? Automated Tests that Always Pass or Fail?

Automated tests are at there most valuable when they catch defects/bugs.

What value do tests that always pass or fail generate? Probably not much - perhaps they even detract! In my experience, the tests slowly become ignored noise due to a constant pass/fail state which will either be attributed to a ‘flakiness’ or a ‘reliable’ feature. Who knows if that is the case.

If your automated tests always pass or fail it is time to review them. An automated test suite that doesn’t catch any defects isn’t useful. Nor is a test suite that cries wolf all the time. There are multiple reasons why this could be the case perhaps the test suite isn’t rigorous enough or asks the wrong questions.

Test Coverage - a Poor Metric for Value

Test coverage doesn’t prove you have valuable tests.

If we agree that valuable tests are those that find defects then tracking test coverage doesn’t tell us anything about value. Test coverage doesn’t tell us if a test catches defects. If I have a project with 100% test coverage does that mean I have a valuable test pack?

We cannot answer this question. We need to dig deeper.

Going Beyond Test Coverage

So if Test Coverage is a poor metric what metrics work well for discovering value? There are many you could use - I have picked out a few I like.

To implement these metrics you will probably need to combine some manual effort with automated reporting. But the results are worth it. You will start to see how valuable your tests are.

Metric 1: False Negative Rate

If you cannot fix a test you must remove it!

Flakiness, flapping tests, unreliable tests - most test suites have them. Tests which for whatever reason sometimes fail. It could be down to multiple things; environment, poor design, strict assertions or a timing issue.

Tracking these failures is very important. You want to see which tests have a high false-negative rate because these tests are detracting value from your suite. They are noise. Once you identify these tests you have two options. Fix or remove them.

Some objections to removal I have come across:

Tests an Important Feature

The unreliable test is important - we need it to test feature X.

An inconsistent test cannot be relied upon. If the test fails do you have confidence in that failure? Does your team? It is much better to remove or redesign the test.

Inconsistent tests will eventually be ignored by the team. You do not want that.

Test Coverage will Drop

Again test coverage doesn’t tell us anything about value. Test coverage is not as important as valuable working tests are.

Metric 2: False Positive Rate

It is better to have a test suite that has failures in its execution than a 100% pass rate.

Worse than an unreliable test is a test that always passes so potentially missing defects. Every time you raise a defect ask the question should the automated tests have caught this? If the answer is yes and the test exists then note it down. Start to track how many times a test misses a defect.

If you have tests that are consistently missing defects you have a problem. Again you should redesign or remove that test. It is giving your team false confidence.

Metric 3: Execution Time

Slow test suite execution slows your team down and detracts value.

How long do your tests take to run? A test suite is most valuable when it provides fast feedback. This allows the team to quickly check work and act on the results. If your team has to wait for an overnight run of tests before release they will be slowed down. When a test suite slows down a team in this way it starts to detract value.

If you see your execution time creeping up then start to think about how you can bring it down. Hopefully, your tests are designed so they can be executed in any order without issues. If that is the case perhaps you could execute tests in parallel?

Metric 4: Confirmed Defects Raised

Tests that find defects are creating value.

This is a simple but powerful metric. Start to record how many defects a test has found. But not just count also record impact and criticality. Start to get a sense of how valuable the defects your tests find are. Use this to demonstrate the value or lack of your test suite brings to the project.

Metric 5: Defect Clustering

Defects often occur in clusters. Tracking these clusters will help you plan your tests.

Every tester has come across defect clustering even if they don’t realise it. Have you ever been working on something and have a gut feeling, this feature always breaks? That is defect clustering at work. Where you find a defect you will often find others and they can be described as a cluster.

Break your project into clusters. This could be done by feature or perhaps repository. Each time a defect is found by your test suite track what cluster it was found in. Start to see where the pattern forms. Use this to guide where you invest your time in a test suite.

Perhaps the Shopping Cart seems to always have issues but after tracking defects in clusters you find it is the Payment logic is the issue. Tracking this way goes one step beyond gut feel. You can prove that the tests you are planning to write will be valuable because you are using data to target troublesome areas of your application.

Writing Tests and Tracking Coverage is Not Enough

It is not enough to just write lots of tests that increase your test coverage metric. We can do better by focusing on tests that find defects. Finding defects is what we want from our test suites after all!

Tracking defects found, the impact of those defects and where they appear you can start to plan tests based on data and provide more value to your team.

Ready for Test - a Symptom of a Poor Quality Culture

2021-01-09T11:31:32+00:00

The Ready for Test column on your JIRA/Agile/Kanban/Whatever process feels safe, doesn’t it? It is a well-known column and on the surface seems to follow nicely with how tickets are worked on. There is a bit of elaboration, then some code is written (hopefully reviewed) and finally we can move it into QA where an independent set of eyes can take a look at it. But doesn’t it all sound a little less Agile and a bit more Waterfall?

Ideas such as Shift Left, Continuous Integration and DevOps are increasingly popular. Their popularity stems from early feedback, sharing knowledge and ownership between team members. Yet Agile software teams continue to treat QA concerns as a separate stage owned by different people. All the benefits of the Agile ways of working above give us are lost. QA team members provide feedback after the task, hoarding knowledge and being passed ownership of quality.

In this post, I want to convince you that in a lot of cases Ready for Test, In QA and other quality gates do more harm than good. They contribute to a culture of separation between QAs and other team members. This separation causes quality to decrease.

Quality is the Concern of All

Let us start with the basics. Everyone on the team has a responsibility to deliver a quality product. That is not something specific to our industry but is true in near all industries. A well-performing team works together and helps each other to achieve a goal whether that is a successful Sprint in IT or on the Athletic track.

So here is the first issue with Ready for Test. If we all have a responsibility to deliver good work then why hasn’t the work been ready for testing from the first step? And why should a specific person check that work?

Testing is more than checking the nuts and bolts fit together. It is also discovering if those nuts and bolts should be put together and if it is being done in the right way.

Testing Early and Continuously

This brings us onto early testing. How can we question design and method early on before we as a team test functionality? As development continues the approach will change, how can we verify this change?

Three Amigos

Get some people in a room - canonically a developer, QA and BA/Product Owner. At the start of a piece of work discuss the aims and the constraints. Talk about the concerns of the ticket be it testing, business or user ones. Note down on the ticket any important points.

You are testing assumptions and the shared understanding of the work. Clearing up misunderstanding early on saves time later on fixing defects and bugs.

Pairing

Once upon a time, I was sat in a retro with my team. I wanted developers on the team to be more available for pairing and working on testing tasks. Some of the developers weren’t too happy. They wanted to focus on feature work. Not “QA” work.

“If we are meant to pair on Test tickets then that means you should be pairing on Dev tickets?”

Yes! Pairing is a great way to test things early and importantly should work both ways.

QAs should pair with developers on feature work asking questions about implementation and improving there understanding. Equally, Developers should be doing the same. We don’t want silos in software development. We want knowledge to be spread across the team so if one person leaves its not a mad dash to take on knowledge transfer tasks.

Discovery / Spike Tickets

Sometimes the team doesn’t know enough about a problem to deliver on the shared promise of quality. In these cases, someone will suggest a Spike or Discovery ticket. A timeboxed assigned task to someone to investigate and return with more information for the team so an informed decision can be made.

These types of tasks are all about testing. Testing an idea, evaluating options and returning a conclusion. They are also a natural fit for Pairing. Devs and QAs should work on these spikes to make sure the correct questions are being asked, the right options are evaluated and results are presented from a multifaceted view.

Manual and Automated Testing - a Separate Stage?

If everyone owns the quality of delivery then everyone owns the verification of that quality right? You cannot have shared ownership of something if only a handful of people verify it. The teams should as a whole be questioning if a suitable level of quality has been achieved before release.

This brings me to testing. A developer doesn’t need a QA to test anything. There is evidence of this already - when the developers on a team write unit tests. This task is managed easily. There is no reason why a developer working on a task cannot also write some functional automated tests. Perhaps they could also do a manual check if required.

I can hear the pushback through my screen. “I shouldn’t test my own work - separation helps find defects.” I agree. But that separation doesn’t need to be from a QA - it certainly doesn’t need to be a separate stage. It should already be happening. Quality is a feature, not an afterthought. Anyone can pair on testing activities and they should be doing so early and throughout the progress of a ticket. Pairing, Three Amigos, Spikes will all allow you to do this.

There is no Ready for Test - It Already is

We have seen techniques to allow us to test early and continuously. If we have been utilising them then what is left? Not much. Testing has been constant, a pair of team members have been working together on a ticket which was well understood because we either held a Three Amigos session to avoid misunderstandings or a Spike.

Ok, there might need to be some final checks driven by how much risk is associated with the work. That is what Demo sessions are for. Demo to your squad, your team, your product owner. A bit nervous before the demo? Grab a team member and demo it to just them - if it makes you feel better to call it pairing!

FAQ

So What is the Role of a QA?

Ok, I have now talked myself out of a job it seems. Sometimes - but more often not.

QAs on a team are typically outnumbered by other team members. They cannot be involved in everything and shouldn’t be. Their time is better spent shepherding the team towards shared ownership of quality and ensuring that collective responsibility is established. Someways this can be done:

Pairing with Developers
Doing Discovery work to Enable Better Testing
Setting up Test Infrastructure to make it easier for others to Test
Liaising with Testers in other teams
Reviewing user feedback. Bringing that information to the team
Running Three Amigos/Retro/Standup sessions
Testing high-risk features where extra reassurance is needed

Where should I put Testing Tasks on the Board?

Treat these tickets like any other ticket. Ready for development, In Progress. Testing tasks don’t need a special column all of there own. It only serves to encourage testing to be the responsibility of a small subset of the team.

Should I Write Consumer-Driven Contract Tests for Services I Own?

2020-07-14T12:31:32+01:00

The Ministry of Testing Slack Channel is free to join and a great place to ask questions of the testing community. A (paraphrased) question I was asked recently:

If I own two services which integrate - do I need to set up a consumer-driven (pact) contract test between them? Perhaps I can use a simple integration test instead?

It is a question I have come across a few times now when working with teams.

TL;DR; Yes you Should

Just because you own both services doesn’t mean the relationship between them stops existing. A Consumer-Driven Contract Test will allow you to keep tabs on that relationship and catch any drift between the two services without expensive, slow and hard to maintain integration tests.

A Quick Recap - What is a Consumer-Driven Contract Test (CDCT)?

I have written before about what a CDCT is but as a quick recap. A CDCT is a contract test where your consumer defines the API contract. The inverse is a more typical contract test relying on the API provider (developer) to define the contract.

It is preferable to have the consumer define the contract. The resultant contract will more accurately reflect API usage and is less likely to go out of date as the contract is generated by the consumer’s test suite.

I like to use Pact as a framework to write these tests but there are other options like Postman.

So that is what a CDCT is. But what is the purpose?

Testing Relationships - The Key to a Successful System

Whether we use integration, end to end or CDCT techniques they are all aiming to do the same thing - test relationships. When we write and execute these tests teams and individuals say things like:

We are checking the system hangs together.

What is meant by the phrase hangs together is the system communicates well together. The Basket service can speak to the Payments service. The Authorisation service verifies that only certain users in the Accounts service can access a resource.

These are relationships and they are critical, without them, you don’t have a system. This is more prevalent in a microservice architecture where the number of relationships increases.

So Back to the Question. Why Integration Tests Can’t Replace CDCTs?

The original question comes in two parts:

Do I need to use a CDCT if I own both services?

Yes

The relationship between your two services still exists when you own the development of them both. The question hints that maybe because both sides of the relationship are owned by the same team that the contract between both services will be easily be maintained - this is not my personal experience.

Imagine a team that owns multiple small well-designed services that all communicate with each other. Like all teams, you are subject to changing requirements and priorities. You spend some time working on one service and then switch to the next. In this scenario, it is very easy for the relationship between just two services to drift.

You or a colleague makes a change in one service which changes to the format of an attribute importantly - let’s say a date field (they are tedious…) well that is easily missed - it is subtle. If you had a CDCT this would be picked up immediately. With an integration test, it might be picked up if you designed the test to check that.

Can an Integration Test Replace a CDCT?

Integration tests can test relationships and CDCTs are not suitable all the time. For example, an external service which will not work on creating CDCTs with you. But in most cases, integration tests are a poor choice for testing relationships - because they are…

Slow

Either because they are waiting for an environment to be spun up or some other downstream system. A CDCT doesn’t wait for any system. It is a decoupled approach to relationship testing meaning you get faster feedback and don’t need to maintain an integration testing environment.

Brittle

The previous example of a date change shows how brittle an integration test can be. Unless your integration test had a specific assert of the date format the issue won’t be picked up. Now I can imagine you saying:

A well-designed integration test will have a format check.

Maybe. But maybe it just checks the date can be parsed rather than a specific format? A CDCT will pick up the format change even if you didn’t specifically write an assert for it (unless you decided to accept any string… then it is on you!).

Integration tests also rely on a specific environment. Either a long-running or temporarily generated environment which inevitably breaks over time (OS & Package updates, config changes) so has to be maintained. CDCT don’t require this. Instead sharing a contract file to support decoupling.

Consumer-Driven Contract Tests > Integration Tests

Integration tests have their place. Sometimes it will be impractical to set up a CDCT because you haven’t got the required cooperation.

But in most cases, I would recommend using a CDCT over an integration test. This advice doesn’t change if you own both sides of the relationship. The same issues as before exist. If anything it makes it easier for you to adopt CDCTs as you don’t need to convince another team!

Selling Teams on Change As a QA

2020-03-02T14:21:31+00:00

How Do I Convince My Team to Do This?

I was asked this question after presenting my talk on Consumer-Driven Contract Tests at London Tester Gathering. It is a great question. People generally don’t like change and even less so when the concept is unfamiliar.

The question went on to ask (paraphrasing) - “How do I approach the Delivery Leads about getting the time to do this?”

That is your first mistake! Delivery Leads, Tech Leads, Product Owners, maybe even QA Managers have different goals to you the QA in the team. Yes we all hopefully want to deliver quality software but approach it in different ways.

The Two Objectives of a Software Project

There are ultimately two objectives of a software project deliver software that is of sufficient quality. The contentious point being quality. What you as a QA consider as ready to release will be different from that of a Lead type role and that is ok.

The primary role of a QA in a team is to champion quality. The primary role of a Lead is to deliver. Of course, both roles care about each point but if you had to pick that is how it typically falls.

So when you ask your Lead Can I have time to investigate this new Idea? they naturally might push back - worried about delivery timescales.

The Secret - Don’t Ask

The best advice I have ever been given was the old adage:

It is easier to ask forgiveness than seek permission. - Grace Hopper…

Guess what - it is true! So below is my approach based on this advice on convincing people of a new idea.

Disclaimer

This is my own personal approach and not all people and companies are the same. Don’t just blindly follow what some stranger says on the internet. You will be taking time away from delivery following this approach so try not to do it around important deadlines. Use your own judgement.

That said you should definitely give this approach a go. You as a QA are the champion of quality so should try new ideas in defending it.

If you find yourself in a place where you are so micromanaged that you cannot have any little side projects in work consider moving on. I have worked in places like that and generally found they are not worth sticking around.

Step 1 - Talk About Your Idea with your Colleagues

My most recent example of this approach is based on my work at John Lewis with Equal Experts. I wanted to introduce Consumer-Driven Contract Tests - but it was met with some skepticism.

So firstly, speak to others who might agree it is a good idea. Discuss the idea with them - ask lots of questions see if it is as good as you think. If even the people you thought would agree do not then think about why. Maybe this is not a good idea.

Try a different angle and refine until someone agrees. You can move onto step two without agreement but the risk of it not working out increases.

Step 2 - Create a Very Small and Simple Proof of Concept

Now you have found some agreement work with it to create a simple working solution to at least part of your problem.

In John Lewis, it was a QA in another team. We worked together to create a very simple Consumer-Driven Contract Test and set up a simple Broker to go with it.

Now you have more than just an idea. You have something you can show people as well. This is ideal for the next stage. It frames the discussion and will help people get over the psychological barrier of taking the first step.

Step 3 - Showcase Your Work

Demo your simple proof of concept to your teams. If you have a related community (such as a weekly QA catch up) demo it there too. Speak to people about it. Get feedback.

As you get the feedback you may find that other people want to be involved. If you do great! This likely means you are solving a problem others have.

In John Lewis after some demo and promotional work, the idea gained momentum. Now there is a post on the John Lewis Engineering Blog.

After doing all this showcase work you should hopefully have a proven out idea to take to your lead.

Step 4 - Speak to Your Lead(s)

Now is the time to ask your Lead for time to investigate this idea further. Show them the proof of concept and how the idea has gathered momentum after you demoed it to your fellow QAs or Developers.

They will of course still be thinking about delivery. But it is a lot harder to dismiss an idea with momentum behind it.

But it Might Not Work Out

At any of the stages above your idea may not work out. Some time was spent but it didn’t help with delivery or quality. That happens in software all the time - not all ideas are winners. But you probably learnt about:

Critical analysis
Technologies
Testing concepts
Presenting to an individual or group
Managing expectations

That seems worth taking the risk to me. But if all else fails ask for forgiveness!

QA Interview Tip - Everything is Not Awesome

2020-02-12T08:18:32+00:00

The Lego Movie Got It Wrong and That is OK

Everything is not awesome (despite the catchy song) and you don’t like every QA tool or technology you have ever worked with and neither do I, at least not completely.

And that is ok. I am not going to judge you in an interview for saying you don’t like a certain tool, tech or technique either partially or completely. In fact, as an interviewer, I often ask interviewees what QA thing they don’t like. I am suspicious when they say nothing (unless you are inexperienced). It is important to question the body of knowledge we have as a community and to constantly evaluate it.

Without questioning what we are doing and why we all risk becoming cargo cult QAs. I want to work with people who are looking for the best way of doing things and not because that is how they have always been done.

A Caveat… Tell Me Why

Please if you don’t like something then tell me why. Just saying I dislike BDD or Generic Unit Test Framework with no follow up doesn’t show me your critical thinking skills. If you don’t like something say why you don’t. It might be buggy, it could be lazily implemented or perhaps it gets in your way.

Things I Don’t Like in QA

In the spirit of this post here are some things I don’t like in the QA world. If you are ever in an interview with me feel free to ask about them. They are all concepts rather than technology for no particular reason.

The Test Pyramid

Simplistic, misused and rarely implemented correctly. The key idea behind the Test Pyramid in fairness is a good one - let’s have most of our tests as tightly scoped as possible. So instead of lots of end to end tests, we have lots of little focused ones instead.

The problem, however, is the pyramid is often supported dogmatically in my experience and rarely implemented well. Instead, teams often try to build up testing in the layers of pyramid rather than thinking about risk what is the highest priority. This, in the end, makes teams feel bad that they never hit the mythical perfect pyramid shape.

Testing is Good. Pyramids are Bad. Ice Cream Cones are the Worst

Integration Tests

I can see the appeal of Integration Tests. Your system is made up of many moving parts and you want to check that they work together. But you don’t want to set up the whole system as it would take too long and is too expensive so you do it in chunks.

The problem with this is the testing in chunks is still expensive and takes too long. Orchestrating your chunks together on an environment can be painful. Setting up test data which avoids conflicts between tests can be worse.

If your system is made up of many services then I prefer consumer driven contract tests with only a very small amount of integration/end to end tests. The small amount of integration/end to end tests that are left act as a sanity check of the very basics of your system. The consumer-driven contract tests verify all the little relationships between the different parts.

BDD Everywhere

You have a suite of Integration or End to End tests and they are written with Selenium. Having read some testing blogs you now use a BDD framework on top like Cucumber because it allows developers and business people alike to read tests in English like language. If your team actually does read the cucumber specs to help with understanding then I don’t have a problem with the technique.

A lot of places don’t do that. They do BDD because someone told them they should. No one ever reads the scenarios when they are completed. It is just an extra layer of complexity in your test framework.

Worse still the scenarios which are meant to be readable by anyone are poorly written. I have seen versions of the below many times:

Given I visit www.google.com
And I check that the 'Search' button is on the page
And I check that the 'Feeling Lucky' button is on the page
And I check the Google Logo is on the page
And I enter the search term 'BDD is fun'
And I click the 'Search Button'
Then I see the Search results page
And The page has results
And they are clickable

That is a mild example (because I got bored) - I have seen many that are much worse.

Why I recommend against using Cucumber

Be Honest and Critical

You are in an interview to show someone two main things - that you are Smart and Get Things Done. (I know there are other skills to display as well…)

Being Honest and Critical about testing trends and tools shows me you are smart. It potentially shows me that you want to get things done too by avoiding doing things for the sake of doing them.

Visualising API Performance Results With GCP BigQuery and Google Sheets

2020-01-15T14:17:49+00:00

API Performance Problems: Lets Capture More Data

The performance tests were failing seemingly randomly. No one on my team knew why our API would suddenly fail on the 99th or 95th percentiles. They wanted easy access to data and visualisations.

We use Gatling for our performance tests and our service runs in Google Cloud Platform (GCP). What if performance statistics from Gatling could be captured and stored in GCP to be visualised using a self-service approach. Turns out they can.

Solution Overview

Gatling stores test statistics in JSON files on disk (more below) those stats can be parsed, stored in GCP Datastore and then using a Datastore backup (in GCP Storage) the results can be loaded into GCP Big Query. From there you can use Google Sheets to load the data into a spreadsheet. Once in Google Sheets, you can do all the usual things you might do with a spreadsheet. Create some pretty graphs, a pie chart maybe.

Datastore --> Storage (Bucket) --> BigQuery --> Google Sheets

That sounds like a lot of moving parts and it is. But the moving parts individually are simple and work together well. The individual tasks are not hard to setup. Let’s start with Gatling.

Gathering Data from Gatling

The first stage is to get the performance stats from Gatling. Starting in the report folder generated by your Gatling test there are two key files that I used for stats:

./js/assertions.json

This file contains the details of all the assertions you make within your test. This is handy as, chances are all the performance stats you care about are in here as your test asserts on them.

JSON Structure

The file contains one JSON object with the following structure:

{
    "simulation": "simulation-name",
    "simulationId": "simulation-id",
    "start": 1576246324329,
    "description": "",
    "scenarios": [
        "scenario1",
        "scenario2"
    ],
    "assertions": [
        {
            "path": "path_to_assertion_target",
            "target": "percentage of successful events",
            "condition": "is greater than",
            "expectedValues": [
                99.9
            ],
            "result": false,
            "message": "path_to_assertion_target: percentage of successful events is greater than 99.9",
            "actualValue": [
                0.0
            ]
        }
    ]
}

./js/global_stats.json

If you don’t assert on the values you want to capture (it is perhaps worth double-checking why you care about them if you don’t assert on them) then your second source of data is global_stats.json. It contains the key stats you might need such as percentiles and response times.

However, one thing to note is the percentile fields follow a Gatling naming convention instead of a clearer 99th, 95th and so on. Use this mapping to help:

Percentile	Gatling Percentile Name
50th	Percentiles 1
75th	Percentiles 2
95th	Percentiles 3
99th	Percentiles 4

Also note the information on groups are setup either using the defaults or by what is in your Gatling config but the naming is consistent regardless (e.g. group1, group2, group3 and group4).

JSON Structure

The file contains one JSON object with multiple sub-objects within it:

{
    "name": "Global Information",
    "numberOfRequests": {
        "total": 1460,
        "ok": 0,
        "ko": 1840
    },
    "minResponseTime": {
        "total": 41,
        "ok": 0,
        "ko": 41
    },
    "maxResponseTime": {
        "total": 3550,
        "ok": 0,
        "ko": 3550
    },
    "meanResponseTime": {
        "total": 146,
        "ok": 0,
        "ko": 146
    },
    "standardDeviation": {
        "total": 365,
        "ok": 0,
        "ko": 365
    },
    "percentiles1": {
        "total": 63,
        "ok": 0,
        "ko": 63
    },
    "percentiles2": {
        "total": 72,
        "ok": 0,
        "ko": 72
    },
    "percentiles3": {
        "total": 761,
        "ok": 0,
        "ko": 761
    },
    "percentiles4": {
        "total": 9999,
        "ok": 0,
        "ko": 9999
    },
    "group1": {
    "name": "t < 55 ms",
    "count": 0,
    "percentage": 0
},
    "group2": {
    "name": "55 ms < t < 75 ms",
    "count": 0,
    "percentage": 0
},
    "group3": {
    "name": "t > 75 ms",
    "count": 0,
    "percentage": 0
},
    "group4": {
    "name": "failed",
    "count": 18191,
    "percentage": 100
},
    "meanNumberOfRequestsPerSecond": {
        "total": 12,
        "ok": 0,
        "ko": 9
    }
}

Pushing Gatling Data into GCP Datastore

With the data found, we can move it into GCP Datastore. Briefly, GCP Datastore is a NoSQL database offered by GCP. It is perfect for storing data like this where we don’t care about modding data relationships and just storing the data.

Personally I used Python to parse the Gatling JSON files and push the data into Datastore. At the time of writing the python packages I used for GCP were:

google-api-core==1.14.3
google-auth==1.8.2
google-cloud-core==1.1.0
google-cloud-datastore==1.10.0
googleapis-common-protos==1.6.0

These things update all the time so check the latest so check pypi package listings for the latest versions to use.

If you are new to Python take a look at Pip and VirtualEnv which makes managing your python packages easy.

It is a bit beyond the scope of this post to teach GCP & Datastore but assuming you have some knowledge of both these topics then this example should get you started:

# Dict containing my stats...
perf_stat = {}
stat_key = "MyStatKey"

# Code to populate dict from JSON files... omitted

# Create, populate and persist an entity
client = datastore.Client(namespace="mynamespace", project="myproject")
stat_key = client.key('MyDataKind')
entity = datastore.Entity(key=stat_key)
entity.update({
    'testRunId': perf_stat['testRunId'],
    'lastUpdated': dt2ts(datetime.now()),
    'testRunStartTimestampUtc': perf_stat['testRunStartTimestampUtc'],
    'global_min_response_time_ms': perf_stat['global_min_response_time'],
    'global_max_response_time_ms': perf_stat['global_max_response_time'],
    'global_mean_response_time_ms': perf_stat['global_mean_response_time'],
    'global_mean_requests_per_second': perf_stat['global_mean_requests_per_second']
})
client.put(entity)

Backing Up the Data to GCP Storage

With the stats captured in Datastore, you could stop here and tell the rest of your team to access the data via GCP Cloud Console. That would work but (particularly with lots of stats captured) the data can be hard to read.

Instead, we are going to load a backup of the data into a bucket using Storage. Using this backup we can load the data into BigQuery and view it using Google Sheets.

Create a GCP Storage Bucket

First, create a Bucket in Storage to store the backup. I also set the retention policy on our Bucket to 7 days because in our implementation this data was loaded into BigQuery as soon as the backup was completed. Once the data is in BigQuery the backup is no longer needed.

Create a Datastore Backup into your Bucket

With the bucket created there are a number of ways to create a datastore backup to it. But I used the command line during our deployment pipeline.

Create a bash file and first make sure you are authenticated using the pipeline:

gcloud auth activate-service-account --key-file=my_key.json

Now add to your bash file the command to start the backup:

# Backup the perf data to a bucket and wait
urlPrefix=$(gcloud datastore export --kinds='MyDataKind' --namespaces='MyNamespace' --format=json gs://my-bucket-gs-url | jq --raw-output '.metadata.outputUrlPrefix')

The key part of that snippet is the usage of jq (which you may need to install). It grabs the first part of the URL returned by the gcloud command which tells you where your backup is stored. You will need that for the next part.

Loading the Data into GCP BigQuery

With the backup now in your bucket here is the final part. Loading the data into BigQuery. BigQuery is a serverless data warehouse solution provided by GCP. It looks very powerful and complicated but for our purpose it is actually quite simple.

First using the GCP Cloud Console navigate to BigQuery and create a dataset and table within it to store your data in.

Update your bash script (the same one with the backup logic) to load the data into BiqQuery:

# Import datastore backup into big data
bq --location=europe-west2 load \
--source_format=DATASTORE_BACKUP \
--replace \
mynamespace:bigquery_dataset_name.bigquery_table_name \
$urlPrefix/namespace_mynamespace/kind_MyDataKind/namespace_mynamespace_kind_MyDatakind.export_metadata

The key things to note in the snippet above:

It uses $urlPrefix defined earlier in your bash script to know where to look for the backup.
The bq command used to load the backup into BigQuery has the --replace flag passed to it. This is important for us as it we never want to append data to BigQuery - just use the latest backup.

With that done your data is in BigQuery and ready to use by Google Sheets.

Bonus! Accessing the GCP BigQuery Data from Google Sheets

Google Sheets can access data stored in BigQuery. It does so using a Data Connector found under the Data menu.

This is a simple wizard to follow. Tell it the dataset and table you want and the query you want to use to get your data and that is it. The data is inserted in a Sheet and you are ready to go.

When you want to get the latest version of the data just hit the Refresh button on the bottom left.

Wrapping Up

By following this post you have (or are perhaps planning to) string together a couple of GCP services in a relatively simple way:

Datastore --> Storage (Bucket) --> BigQuery --> Google Sheets

The reward for this plumbing work is a more friendly way to allow your whole team to access performance data. Everyone on your team is probably familiar with spreadsheets. The less technical members of your team don’t need to understand the other services sitting underneath delivering this information to the sheet. Finally, it can all be orchestrated via your deployment pipeline.

It is still early days for my team but so far letting them manipulate performance data themselves has proven quite useful. It is self-service and they can create any number of visualisations they like using a familiar tool.

Introduction to Automated Security Testing with OWASP Zap, Dependency Checker and Glue.

2019-04-09T17:44:49+01:00

Security testing can be really time-consuming. Ever tried to organise a penetration test for your website? It is expensive! For my current client, we wanted to think about how much security testing can be done ahead of time in an automated way. Not as a replacement for professional penetration testing but as a way to give us some confidence before that stage that we are catching issues as early as we can.

I did this by adding automated security tests for common issues in our codebase. For example, insecure dependencies or API endpoints that are vulnerable to SQL injection attempts. The Open Web Application Security Project (OWASP) has a couple of tools that can help with this. OWASP Dependency Checker, ZAP and Glue. In this post, I will introduce them all and how they are meant to work together.

Dependency Checker - Verifying Code you Consume

Dependency checker does what the name suggests - it checks your dependencies looking for ones with known vulnerabilities. Once it finds some it will let you know what they are a link to the National Vulnerability Database (NVD). It is fast, easy to set up and gives you some confidence that your code is built on a solid foundation.

Checking your dependencies is one of the first things you should do when looking to add automated verification of your application’s security - it is a quick win! Check the paper Unfortunate Reality of Insecure Libraries to understand more about the scale of the problem.

ZAP - Verifying Code you Write

With your dependencies checked what about your own code. Zed Attack Proxy (ZAP) is the tool for this. It checks your code by doing both passive and active attacks against your site or API. Passive attacks are things like information disclosure such as your server version in the headers. In contrast, active attacks directly attack your application using attack vectors such as SQL injection.

One thing to keep in mind is the Active attacks can cause your site issues - after all, they are malicious requests! So it is best to run this tool against a test environment rather than the live site.

Glue - Bringing Results Together

The two tools above will produce plenty of results to look at. But like all security testing tools, they produce false positives. Without a clear way to manage this, it can become painful quickly. Glue helps with this by processing the output of Dependency Checker and Zap and publishing the results somewhere for you to review and if need be mark issues as false positives.

For example, in my setup, I set up glue to interact with the JIRA project for my team. Now once the security tests are completed the issues found are automatically raised in JIRA where engineers can review them. By doing this we make sure test results are addressed and any false positives can be marked as such.

Glue will even run a lot of tools for you. Although I had more success with Dynamic Tasks and would recommend them instead of using the built-in ones.

Summary

Using the three tools together as part of your CI pipeline gives you the ability to address security issues sooner than you might usually. Although it is important to remember that all security tools report false positives and do not catch everything so in my opinion these tools are best used to complement a professional security review for your project. But if you haven’t the budget for one of them then these tools might help you get some confidence!

Quickstart: Try Static Analysis with SonarQube and Docker

2018-11-06T14:51:34+00:00

Recently I started a new contract and was in the rare position of joining a team before the developers! Without a team producing work, I wanted to think about ways to get the team off to a good start. My last post was about static analysis with Sonarqube. I love static analysis tools they are like an additional tester in your team and when the results are taken in the context of the wider effort they can be really valuable. Below I want to show you how to set up a quick proof of concept Sonarqube server to discuss with your team.

Overview

Before starting this guide I assume you have a superficial level of Docker knowledge (I am not an expert) and you have it installed. There are three steps to getting started in total it will probably take 15 minutes (excluding download times):

Create a docker image with SonarQube installed using Docker Hub
Run the docker image just created.
Install Sonar-Scanner and create a SonarQube properties file in the root of your project to run a scan.

Create a SonarQube Docker Container

To use SonarQube you need to be running the server somewhere. We will use Docker. It will download a docker image which contains SonarQube for us already configured and set it up as a container on your machine. To do this open your command window and type:

docker run -d --name sonarqube -p 9000:9000 -p 9092:9092 sonarqube

The output of this command should look similar to:

Unable to find image 'sonarqube:latest' locally
latest: Pulling from library/sonarqube
bc9ab73e5b14: Pull complete 
193a6306c92a: Pull complete 
e5c3f8c317dc: Pull complete 
a587a86c9dcb: Pull complete 
a4c7ee7ef122: Pull complete 
a7c0dad691e9: Pull complete 
367a6a68b113: Pull complete 
60c0e52d1ec2: Pull complete 
c9d22bc43935: Pull complete 
884af0bfbb9a: Pull complete 
35a8cd0c916a: Pull complete 
9f9ecbe7a343: Pull complete 
af800bded4f3: Pull complete 
Digest: sha256:cc57b262ee9e7145456dee8c7ae24622c82b22cabeaac4651e7dd642da806f2e
Status: Downloaded newer image for sonarqube:latest
a263b203864adc366919ba9cde3cde87542c96046af7a8b9d7ebc1f155ec2204

This will download the latest SonarQube image from Docker Hub and will set up a container using it. To start your server you just need to run this container.

Run the SonarQube Docker Container

Now if you run: docker container ls in your command window you should see your container named sonarqube - for example:

My-MacBook-Pro:kotlin thomas$ docker container ls
CONTAINER ID   IMAGE       COMMAND         CREATED           STATUS            PORTS                                            NAMES
68be74d0aa51   sonarqube   "./bin/run.sh"  About an hour ago Up About an hour  0.0.0.0:9000->9000/tcp, 0.0.0.0:9092->9092/tcp   gracious_noyce

Now you are ready to start the SonarQube server using the docker run sonarqube command. If this was successful you should see your command window fill with text about the server starting and once loaded be able to see the dashboard at http://localhost:9000. Now you can scan your project.

Scan your Project with Sonar-Scanner

To have some results in your proof of concept SonarQube server you need to scan a project. Sonarqube supports a host of languages so pick a project written in a supported language (you may need to install a plugin). Next, make sure you have the Sonar-Scanner application which you can run from your command window. But before you run it at the root of your project add a sonar-project.properties file. This will tell the scanner a bit about your project and where to send the results. Below is an example of the minimum required information:

#----- SonarQube server
sonar.host.url=http://localhost:9000

#----- Project Key
sonar.projectKey=4a27fa8c666747f6956d75ae63fb24b9

#----- Project Name
sonar.projectName=MyProjectName

#----- Project Version
sonar.projectVersion=1.0

#----- Source files (relative)
sonar.sources=.

The projectKey needs to be unique to your project on the SonarQube server and the rest is self-explanatory. With all this in place with your command line in the root of the project where the sonar-scanner is located (unless you have installed it in another way such as with Homebrew) enter sonar-scanner and press enter. You will see an output like this:

My-MacBook-Pro:kotlin thomas$ sonar-scanner
<Redacted for brevity>
INFO: ANALYSIS SUCCESSFUL, you can browse http://localhost:9000/dashboard/index/4a27fa8c666747f6956d75ae63fb24b9
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at http://localhost:9000/api/ce/task?id=AWbpY5h5xoaTc7Huvu3D
INFO: Task total time: 4.403 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 5.628s
INFO: Final Memory: 11M/50M
INFO: ------------------------------------------------------------------------

Once completed your results will be in the SonarQube dashboard by following the link. Now start to play around with SonarQube and your team’s projects!

Static Analysis for .NET Core Projects using SonarQube

2018-03-21T13:54:00+00:00

Static analysis is a way of automatically analysing code without executing it. As a development team, this is really powerful as once the static analysis software is up, running and integrated with your deployment pipelines you can gain an extra tester in your team with little ongoing maintenance! While some of the issues static analysis software finds are not always high value (code styling for example) some are issues your engineers are less likely to notice such as obscure security flaws and out of date dependencies.

SonarQube; A Static Analysis Tool

SonarQube is a static analysis tool that I have been using and evangelising at Just Eat. Some of the issues it finds:

Logic Bugs
- Example: Checking for null on an object that is always null.
Security Vulnerabilities
- Example: Passwords in the source code
Code Smells
- Example: Hardcoded URIs and Methods with a high Cyclomatic Complexity score.
Test Coverage
- SonarQube, however, does not calculate this by itself but instead ingests test reports from your CI build.
Code Duplication

This information is all collated and displayed in a central dashboard for your project which gives you an overall view of the “quality” of your project. While some of the issues above might seem trivial (the code smells category is often guilty of this) even in these cases, monitoring the trivial metrics to see if they deteriorate over time is a useful measure for standards dropping within your team.

Quality Gates

Related to dropping standards are what SonarQube calls Quality Gates. Using these you can set triggers to throw either warnings or errors in your CI builds based on rules set in your Quality Gate. Example rules include:

Code Coverage below X%
Bug Count increased by X from the last build
Vulnerability Count increased by X from the last build

If any of these rules are broken you can choose to throw warnings or errors during your build process. Sonarqube can even be set up to break your build. But in my opinion, this isn’t a good idea.

The Case Against Breaking the Build

BuildBreaker is an official SonarQube plugin which you can use to break your CI builds if a Quality Gate triggers any errors. On the face of it, this makes sense. You as a QA want your developers to maintain high standards so stopping them from breaking the Quality Gate rules by stopping there build from succeeding is an easy way to do this.

But if only it were that simple. Setting up SonarQube in this way has some key disadvantages:

It encourages the often observed us vs them dynamic between developer and QA.
In continuous delivery environments, you are now holding up a build. In the best case, the Quality Gate has detected a serious drop in quality or worse case trivial issue(s) have caused the build to fail. This won’t make QA popular.
Breaking the build is likely to discourage developer buy-in. Without buy-in from your team, the issues found by SonarQube will never be resolved.

SonarQube should be used as a passive monitor of the health of your projects. By using it in this way you can start to work with your team rather than against them and use the metrics SonarQube discovers to start having conversations about how things might improve and even get involved in improving them yourself! SonarQube themselves did a blog post “Why You Shouldn’t Use Build Breaker” with similar opinions.

Github Integration

The final feature of SonarQube you should be aware of is it’s GitHub integration. With the Sonar-Github plugin, you can set up integration with Github to comment on Pull Requests (PR) with issues found by SonarQube in the changes during your CI builds.

One issue to note is you will probably want to differentiate between PR builds and builds of your master branch.

SonarQube keeps a track of your latest build results to compare the next build to for purpose of its Quality Gate. However, for PR builds you probably don’t want the results stored as you want to always compare the PR build to latest results from _master _otherwise a developer could just rebuild their branch in CI and the new issues previously found would no longer be new and disappear! Below is a PowerShell script I have used in teams to do this differentiation between PR and master branches:

$github_branch_refs_parts = "%teamcity.build.branch%" -split "/"
$is_pr = ($github_branch_refs_parts.Count -eq 2 -and $github_branch_refs_parts[1] -eq "merge")

if (-Not $is_pr) {
    Write-Host "Running SonarQube in master branch mode"    
    SonarQube.Scanner.MSBuild.exe begin /k:"%sonar.project%" /d:"sonar.host.url=%sonar.host.url%" /d:sonar.cs.dotcover.reportsPaths="dotCover.html" /v:"%build.number%"
} else {
    Write-Host "Running SonarQube in PR Mode"

    $pull_request_number = $github_branch_refs_parts[0]
    $repo = "%repo_owner%" + "/" + "%name%"

    $command = 'SonarQube.Scanner.MSBuild.exe' +
    ' begin' + 
    ' /k:' + '%sonar_project%' + 
    ' /v:' + "%build_number%" +
    ' /d:sonar.host.url=' + "%sonar_hosturl%" +
    ' /d:sonar.github.pullRequest=' + $pull_request_number +
    ' /d:sonar.github.repository=' + $repo +
    ' /d:sonar.github.oauth=' + "%sonarqube_github_oauth_token%" +
    ' /d:sonar.analysis.mode=' + "preview" +
    ' /d:sonar.scanAllFiles=' + "true" +
    ' /d:sonar.github.endpoint=' + "%github_api_endpoint%"

    Write-Host $command 
    Invoke-Expression $command
}

The key difference between the two SonarQube.Scanner.MSBuild.exe commands is in the analysis mode Preview the results are not saved by the SonarQube server so the comparison is always done against the latest master build instead.

Analysis Mode Preview Deprecated in SonarQube 6.6

The Preview option is being deprecated according to the documentation, however, I have not seen an alternative proposed yet so cannot suggest how you might handle this in SonarQube 6.6+.

Setup SonarQube with a .NET Core Project

SonarQube now supports .NET Core as of SonarQube Scanner for MSBuild version 2.3. So the setup is actually quite simple:

Install the SonarQube Scanner on your CI server. I have used a build step for this and used Chocolatey to manage the package installation.
Before running your build step add another step which calls SonarQube.Scanner.MSBuild.exe begin so SonarQube can setup hooks into your projects build process. Using the script above might be a good starting point.
After the build process is completed end your SonarQube scan. This can be done with:
> SonarQube.Scanner.MSBuild.exe end
During the end step the SonarQube scanner will upload a report to the SonarQube server and your projects dashboard will be updated.

And that is it! Well almost…

Setting Up a Project Guid

For SonarQube to recognise your project from other projects it needs an identifier. The development team decided to use the ProjectGuid property in your .csproj file. You might not have one as they are not required by default. So double check your csproj files and update them to include a ProjectGuid property. If you miss this you will see the following error in your CI logs:

WARNING: The following projects do not have a valid ProjectGuid and were not built using a valid solution (.sln) thus will be skipped from analysis…

If you see that warning add your ProjectGuid!

Contract Testing with Pact in .NET Core

2018-03-13T17:07:18+00:00

When working in a microservice architecture it can be hard to verify the whole system end to end due to all the moving parts involved. Often the purported solution to this is to write integration tests which verify a couple bits of the system at the same time with the test mocked out. If all these subsections of the system pass their respective integration tests we can be confident in the system, right?

The Problem with Integration Tests

Integration tests are a good way of verifying our system as they use real (not mocked out) components but quite a lot can go wrong. Integration tests are:

Unstable

Much like the end to end tests, integration tests take a lot of effort to keep up to date and are often flakey and unstable. And unstable tests are worse than no tests as they train your team to ignore test results.

Broadly Scoped & Unspecific

Think about when you are writing some code - one of the first design practices you might have learnt is to keep scope tight. For example, if the code in a loop needs the variable then declare it in there so the code outside of the loop cannot access that variable. That way if the code has a bug in the looping section you know that variable can only have been modified by code within the loop.

A tight scope should also apply to your tests. When your tests fail you should be able to quickly pinpoint where the failure happened quickly and start to understand why. This is why developers and QAs love unit tests as they are specific and tightly scoped when unit tests fail you typically know exactly why and how to go about developing a fix. This isn’t the case with integration tests when they fail it could be a whole host of things! Perhaps it is the first service, maybe the third? What about datastores did we mock them out? Were those mocks correct?

Speed

Integration tests are just slow. It is slow to spin up the subset of your system under test. It is slow to run the tests against the system under test. If you want speed in your test suite the integration tests won’t help.

Expensive

All that time trying to keep your integration tests up to date and fast comes at a cost. Developers or QAs will need to spend quite a lot of time keeping the tests green - in the same way, they keep the end to end tests green.

An Alternative: Contract Testing

Instead of creating unstable, broadly scoped, slow and expensive integration tests there is an alternative, contract testing. Contract testing works in a different way to integration testing. Instead of testing real interactions of a subset of the system contract tests generally work by thinking of a data provider and a data consumer:

The Providing API has a suite of unit tests referred to as contract tests which test that for different API calls to itself the API returns the expected data.
The Consuming APIs don’t have integration tests with the Providing API instead they trust(!) that the Providing API will return the data as they expect it. That they will honour the contract they have with the consumer API.
As long as the Providing API keeps there contract tests up to date and _communicates _with the teams looking after the Consuming APIs everything will be fine.

Nice But Unrealistic

The three steps above are idealistic. Even with the best intentions providing API teams will struggle to keep consuming API teams up to date. What is needed is a way for the teams to keep themselves up to date as much as possible using inter-team interaction to talk about changes they discover themselves.

More Realistic Contract Testing: Pact.io

Instead of relying on teams to keep each other up to date what if the computer did most of the work. This is the problem that the Pact.io testing framework solves which makes contract testing more realistic. Pact follows an updated flow:

The Consuming API has a suite of component tests referred to as _pact tests _which use Pact for mocking out calls to the Providing API. Once the test run is completed a PactFile is created. This is a JSON record of all the mocked out requests and responses the Consuming API made to the Provider API.
The Providing API has a suite of unit tests which consume this PactFile generated by the Consuming API and replay the requests against itself and check the responses match the actual responses from the Providing API.
If the Providing API’s pact tests fail that teams build goes red and then they can speak to the Consuming API team to see what mocks they are using and if they are correct or not.

Now communication is driven by failing tests and it is the Consuming API team which are driving the tests for the Providing API team. This approach makes it much more likely that the Providing API team will speak to the Consuming API team as without speaking to them about changes and updates for there testing the Providing API pact tests will fail and there build will break.

Pact Test Implementation with .NET Core

Pact has been implemented in many languages including .NET Core by the Pact-Net project on Github! So you can start to write pact tests on your .NET Core projects. To get started you can check their examples and if you are ready to check out my Workshop on Github.

Learn More

Verifying Microservice Integrations with Contract Testing (Video)
Pact Docs: Convince Me!
Contract Testing with Pact (Prezi)

Tom Driven Dev

Going Beyond Test Coverage

What is Value? Automated Tests that Always Pass or Fail?

Test Coverage - a Poor Metric for Value

Going Beyond Test Coverage

Metric 1: False Negative Rate

Tests an Important Feature

Test Coverage will Drop

Metric 2: False Positive Rate

Metric 3: Execution Time

Metric 4: Confirmed Defects Raised

Metric 5: Defect Clustering

Writing Tests and Tracking Coverage is Not Enough

Ready for Test - a Symptom of a Poor Quality Culture

Quality is the Concern of All

Testing Early and Continuously

Three Amigos

Pairing

Discovery / Spike Tickets

Manual and Automated Testing - a Separate Stage?

There is no Ready for Test - It Already is

FAQ

So What is the Role of a QA?

Where should I put Testing Tasks on the Board?

Should I Write Consumer-Driven Contract Tests for Services I Own?

TL;DR; Yes you Should

A Quick Recap - What is a Consumer-Driven Contract Test (CDCT)?

Testing Relationships - The Key to a Successful System

So Back to the Question. Why Integration Tests Can’t Replace CDCTs?

Do I need to use a CDCT if I own both services?

Can an Integration Test Replace a CDCT?

Slow

Brittle

Consumer-Driven Contract Tests > Integration Tests

Selling Teams on Change As a QA

How Do I Convince My Team to Do This?

The Two Objectives of a Software Project

The Secret - Don’t Ask

Disclaimer

Step 1 - Talk About Your Idea with your Colleagues

Step 2 - Create a Very Small and Simple Proof of Concept

Step 3 - Showcase Your Work

Step 4 - Speak to Your Lead(s)

But it Might Not Work Out

QA Interview Tip - Everything is Not Awesome

The Lego Movie Got It Wrong and That is OK

A Caveat… Tell Me Why

Things I Don’t Like in QA

The Test Pyramid

Read More

Integration Tests

Read More

BDD Everywhere

Read More

Be Honest and Critical

Visualising API Performance Results With GCP BigQuery and Google Sheets

API Performance Problems: Lets Capture More Data

Solution Overview

Gathering Data from Gatling

./js/assertions.json

JSON Structure

./js/global_stats.json

JSON Structure

Pushing Gatling Data into GCP Datastore

Backing Up the Data to GCP Storage

Create a GCP Storage Bucket

Create a Datastore Backup into your Bucket

Loading the Data into GCP BigQuery

Bonus! Accessing the GCP BigQuery Data from Google Sheets

Wrapping Up

Introduction to Automated Security Testing with OWASP Zap, Dependency Checker and Glue.

Dependency Checker - Verifying Code you Consume

ZAP - Verifying Code you Write

Glue - Bringing Results Together

Summary

Quickstart: Try Static Analysis with SonarQube and Docker

Overview

Create a SonarQube Docker Container

Run the SonarQube Docker Container

Scan your Project with Sonar-Scanner