Before any data analysis, many business teams forget or underestimate the work of data cleaning. In its 2020 Magic Quadrant for Data Quality Solutions, Gartner put the average annual cost of poor data quality at $12.9 million per organisation. In most cases, poor data quality generally comes from human errors like duplicate records of clients or missing values.
Poor data quality also causes a damaging impact on AI strategy – organisations are unable to develop AI tools and deploy them at large scale because, for AI to perform, they need to rely on reliable data.
It is in this context that I built a small app that takes a CSV file, removes duplicate rows and handles missing values, and lets you download the cleaned result.
What does the app's libraries do?
The app is made with Streamlit, an open-source Python library that turns scripts into web interfaces without requiring a full front-end. The code stays linear and easy to follow — a script, read from top to bottom, rather than a set of routes and templates. It also works natively with pandas: a single call to st.dataframe() renders a DataFrame as a sortable table, which makes it straightforward to show the data before and after cleaning.
The cleaning itself relies on pandas, the standard Python library for working with tables. Nearly every operation the app performs — dropping duplicates, handling blanks, converting types — is a single method call.
What does the app do?
Once launched, the app can remove duplicate rows and empty records from an uploaded CSV file. All you need to do is upload the file you want to get cleaned. The only limit is the size, which is fixed to 200MB.
At first, the uploaded file is displayed on the web interface, meaning you can take a look at the existing records before taking any action to clean the file.
A checkbox then appears, suggesting you remove duplicates and NONE values. A couple of seconds later, the table is once again displayed, but without errors, and you have the opportunity to download, with a simple click, the file.
Why data cleaning's process is crucial before any analysis task?
The reason is straightforward. A duplicated record does not announce itself — it simply counts twice. Every average, every total, every segment share shifts slightly, and nothing in the output indicates that anything is wrong. The analysis runs perfectly well. It just describes a population that does not exist.
Furthermore, as mentioned above, AI systems are only as good as the data behind them. Wrong inputs rarely produce obviously wrong outputs — they produce plausible ones, at scale, with no indication that anything went wrong.
Finally, in an economy where data is treated as a strategic asset, its quality determines its worth. Most organisations rely on data to make their strategic decisions for the future and develop their activity.
Where This Goes Next?
With Atlas, I approach each project as a product rather than a demonstration — something that is updated, extended, and revisited rather than published once and left alone.
For example, I am thinking about uploading .xls compatibility, as many databases are from Microsoft Excel files. In addition, why not add features that allow any user to update the type of data string for better analysis, such as transforming a string to data format.
In the meantime, feel free to download the github repo at the following address: https://github.com/flohms18/streamlit-data-cleaning or test it online: https://flohms18-streamlit-data-cleaning-app-cjejm4.streamlit.app/
Don't hesitate to reach me on LinkedIn (link on my home page) if you have any questions or comments!