FACTS AND DIMENSIONS Contact Careers Clients Data Quality Catalogue SQL Training ☰

SQL Training Catalogue Data Quality Clients Careers Contact

DATA QUALITY

Data quality in practice

When it comes to gathering the data your organisation needs, automation will likely be your first instinct.

Data sources can often be less than accurate. Let us deal with the inconsistencies, and the contact with the data providers, leaving you to concentrate on the data.

Here are some of the inconsistencies we have encountered:

A duplicated pair of values found in RTT.

These dates are inconsistent between the values in the rows, the title of the document, and the filename.

This spreadsheet contains a formula which fails to incorporate every data point. The sum should extend to cell F42380.

Dates in many datasets are transposed, or misinterpretted as different date formats.

The title states Q2, but the quarter column states Q1.

Here we have an example of a misspelt filename.

An integer value has been recorded with erroneous decimal places.

Year headings do not match the values in the table, or the sheet name.

These superscript numbers, in brackets, should not be misinterpreted as data.

When a CSV is prepared or modified in Excel, certain strings including numbers can be misinterpreted as floating point numbers.

This field is titled prevalence in other spreadsheets, but inconsistently titled calculation here.

These headings, from PROMs Key Data, are completely transposed.

Duplication of data within a spreadsheet can result in incorrect sum total values.

A spelling error in a heading, can lead to a spelling error in a field name.

Previous releases of this data contained a null value for the totals code field. This release featured a hyphen.

This file contains duplicated ODS codes alongside incorrect location name information.

Web scraping can introduce errors with misspellings such as this.

This spreadsheet has been formatted to only show the correct whole number, but the data contained in the cell, incorrectly has decimal places.

Between one data release and the next, the definition of this column has changed.

In this spreadsheet a the Number of Adult critical care beds, is reported as being 15.5.

Theses Diagnostic ID numbers have been set to 0.

There are true values, there are false values, and then there's this!

An integer value has been recorded with erroneous decimal places.

If a textual date has been used, and then misspelt, an automated ETL system will error.

This spreadsheet contains cells which have white text, effectively making the data invisible.

Another date mismatch between the filename and the file contents.

This spreadsheet has been created with a heading in column H, but the values for that column placed in column G.

When a CSV is prepared or modified in Excel, certain strings including numbers can be misinterpreted as floating point numbers.

The filename is for Quarter 1, but the sheet states Quarter 3.

The field headings here should state Org_code and Organisation_name.

The filename states April, the document Introduction sheet states May.

In an Excel spreadsheet, there can be hidden columns, causing sums of columns to be mismatched from the visible data.

CL is in the headings, where it should be CI.

This data contains a date and month separated by 2 spaces.

This CSV contains unquoted text, which contains commas, hence separating the columns incorrectly.

Some data sets take a great deal of effort to process into a usable format.

This CSV contains repeated headings in the middle of the data.

The filename suggests a date range covering July 2011 to 2012. The data shows just July 2011.

The total stated in the column (as text), disagrees with the total calculated by Excel.

Visit our CCS profile

Cyber Essentials certified

YouTube

Get access to all the data via your browser, free, for 30 days with Snowflake