Measure the Quality of your Source Code!

Quality is something which is hard to define. Different areas of expertise use their own definitions of what quality is. Without an objective standard which carries weight, anyone can claim to provide a quality product or service according to some standard. This makes it difficult to compare products and to formalize which characteristics a product or service needs to have. In this blog post I’ll provide an introduction to ISO/IEC-5055 which is a quality standard which allows us to measure the structural quality of source code objectively.

Standards organizations

The ISO is an international standards organization which has the authority to create standards which carry weight. The members of ISO are national standards institutes. ISO itself is not government owned and independent. The standards ISO creates however cover a large number of areas. Specifically for electrical, electronic and information technologies, they often work together with the IEC in a Joint Technical Committee (JTC). The IEC also creates standards and they don’t want their standards to overlap or compete with each other. By working together in a JTC, they can prevent that.

Part of the standards related to quality have been proposed by the CISQ (Consortium for IT Software Quality). CISQ is a consortium founded by, among others, the OMG (the Object Management Group). The OMG is known for standards like CORBA, UML and BPMN. The members of CISQ are private sector companies. Mostly companies who have interest in a shared definition of what quality is. The OMG as part of CISQ managed to get the proposed quality standards through to be promoted to ISO/IEC standards.

What is quality?

The term quality is defined in ISO 9000 (criteria for a quality management system) as the degree to which a set of inherent characteristics of an object fulfills requirements.

Note that:

• “degree” implies quality is a variable
• “a set of” implies quality is not a single characteristic
• “inherent” as opposed to “assigned”, means existing in the object therefore neither price nor delivery is a quality characteristic of a product but can be a characteristic of a service
• “object” means anything perceivable or conceivable therefore the term quality can be used relative to both tangible and intangible objects e.g. the quality of commitment.
• “requirement” means a need or expectation that is stated, generally implied or obligatory

Why measure quality?

There are a lot of reasons why measuring quality can be beneficial. If you measure quality in an objective manner, you are able to compare products, modules and by proxy, the people or companies producing them. You can apply quality standards directly in contracts. You can use quality measures as a way to promote your company and products. Measuring quality can help prevent weaknesses and maybe even prevent product liability cases. 

For developers, it helps to know if you make choices in your code which can lead to lower quality (think performance, security, stability, maintainability). If you do not measure it, you could be unaware if you make wrong choices. Visualizing code quality is also a great motivator. You can try and become a better programmer by providing measurable better code.

ISO/IEC 25010

ISO/IEC 25010 defines a standard for evaluating system and software product quality. Part of this standard are the definition of quality characteristics.

These characteristics however are not concrete, not measurable. Enter ISO/IEC 25023. This standard provides concrete measurement of system and software product quality. However when looking at quality of source code, this standard cannot be used directly. The following study elaborates on that in more detail. There are two main reasons you cannot apply the measures easily to source code;

• The measures are not compatible with each other and cannot be used to obtain a single number indicating quality. For example, part of reliability is availability. This is measured by mean down time. Another aspect of reliability is recoverability. Recoverability is measured by mean recovery time. These two measures (together with the other aspects of reliability) cannot be summed to create a single measure for reliability.
• The measures are mostly behavioral. They are measures which apply to a running application which is in use and not so much to the source code.

ISO/IEC 5055: Automated source code quality measures

ISO/IEC 5055 complies to ISO/IEC 25010 and expands ISO/IEC 25023 by defining measures which are applicable to source code. It is a freely available standard which you can download here. It has been published in April 2021 and is thus a relatively recent standard.

An important reasons why ISO-5055 was created is IoT. Software is used by IoT devices usually contain code which is difficult to update once weaknesses are discovered. For example updating the device requires manual actions at the location of the device. Since it is expensive to fix issues (especially in IoT cases), it helps if you can detect quality issues early during development, preferable even in a development environment, before official tests have been conducted or a release is made. This saves a lot of time, effort and thus costs.

ISO/IEC 5055 defines weaknesses and detection patterns which allow automation. The weaknesses themselves are registered in the CWE (Common Weakness Enumeration) database which is maintained by MITRE. MITRE is the same organisation who also maintain the CVE (Common Vulnerabilities and Exposures) database. Usually CVEs can be linked to CWEs. CWEs are usually less specific (often do not specify a concrete product or framework). Initially the CWE database only contained security related information but this has been expanded to also contain reliability, performance and maintainability issues. You can browse the ISO/IEC-5055 weaknesses here.

The weaknesses which are described in ISO/IEC-5055 have been evaluated by an international team of experts and identified as requiring remediation. ISO/IEC-5055 contains 138 weaknesses. Because it is a relatively short list, it can relatively easily be automated.

The weaknesses cover different layers of an architecture. From unit to layer to system. An important reason for this is that applications nowadays, mostly cannot be looked at in isolation as a single black box but are part of a larger network of interacting components which also have an internal structure.

Below are some examples of measures at different levels;

Tools

While ISO/IEC 5055 provides a definition of weaknesses and detection patterns, manually checking them in for example a code review, is inaccurate, time consuming and boring. Since the detection patterns are formalized, they can be automated. ISO/IEC 5055 is a relatively new standard (April 2021) so not many tools are currently 100% compliant, but that number is expected to grow. The Executive Director of CISQ is also Chief Scientist at CAST (Dr. Bill Curtis). CAST ‘MRI for Software ‘is the first tool which is claimed to be compliant (here).

Gartner is a research and advisory company which provides (among others) comparisons of software in so-called magic quadrants. Below you can see the magic quadrant of the ‘Application Security Testing’ category of the past 3 years. This category overlaps with the weaknesses which are described in ISO/IEC 5055 (completely with the security weaknesses and partially with the other weaknesses).

As you can see, the steady leaders over the past 3 years have been Checkmarx, Veracode, Synopsys and Micro Focus. When looking at their reported coverage of CWE weaknesses (for Checkmarx I could not find this data) and put them next to the ISO/IEC 5055 weaknesses, you can get coverage percentages like;

• Synopsys (29%)
• Veracode (18%)
• Micro Focus (8%)

This is not a completely honest comparison though. CWE is layered and has categories. When all the CWEs under a category are covered, you can also count the category as covered, but I did not include logic for that in my comparison. I also did not look at language coverage. 

The tools mentioned in the magic quadrant are commercial tools. When looking at open source tools which can be used for Java, for example here, you can conclude that tools usually have a certain area of focus. For example (more general) code quality/standards or security. A tool usually covers quite some CWEs in their focus category but less in others. 

For Java you can look at tools like Checkstyle, PMD, Spotbugs, FindSecBugs, the OWASP Dependency-Check and  Anchore Engine. All these tools provide static scanning capabilities (covering several of the ISO/IEC 5055 weaknesses) and do not focus on running applications. Not all tools provide a direct mention of which CWEs are covered. This makes determining ISO/IEC 5055 coverage difficult. A combination of tools provides the best coverage overall.

It is not difficult to start making small steps in measuring quality of your own source code!

References

• The ISO/IEC 5055 standard
  https://standards.iso.org/ittf/PubliclyAvailableStandards/c080623_ISO_IEC_5055_2021(E).zip
• CISQ webinar on ISO/IEC 5055
  https://www.it-cisq.org/webinars/iso-5055-automated-source-code-quality-measures.htm
• CISQ ISO/IEC 5055 weaknesses as documented by MITRE
  https://cwe.mitre.org/data/definitions/1305.html
• CWE coverage of open source tools
  https://codedx.com/blog/a-look-at-cwe-coverage-across-open-source-and-commercial-static-analysis-tools/
• CAST MRI for Software
  https://www.castsoftware.com/products/MRI-for-Software