Data Management Strategy: A Kaizen Approach
“Take time to improve our data management processes? Sorry, we are just too busy”… fixing errors from broken data processes.
This refrain is more common than you think in IT departments of all sizes. Or maybe you live that reality every day and are fully aware that clunky, error-laden processes eat away at your team’s efficiency (and morale).
Data management and continuous improvement may sound like they should always go together, but they often don’t.
Many times our data management practices involve too many business-critical data processes that break regularly and need to be improved, but we have no time to make the needed improvements because there are so many data processes that need to be “managed” (a.k.a. remediated regularly).
So, how can we break this cycle?
Welcome back to the second of three posts on how to refine your strategy for Data Lifecycle Management (DLM)!
In this post, we will focus on Data Management as the second of the three DLM stages: Data Collection, Data Management, and Data Deletion.
Kaizen for Data Management
The Kaizen approach, famously championed by the Toyota corporation, suggests that small organizational changes can lead to a culture of continuous improvement. This culture will ultimately lead to better processes, greater efficiency, improved outcomes, and increased morale.
The Kaizen Institute states,
“As part of the corporate culture, continuous improvement becomes an ongoing process integrated into the organization’s daily activities. Employees are encouraged to challenge the status quo, suggest ideas, and implement improvements. Continuous learning and development are valued, and mistakes are seen as growth opportunities.”
This means that adopting a Kaizen approach to your data management strategy can be a lever for driving a continuous improvement culture on your data team without sweeping, drastic changes.
Small improvements to existing processes can slowly bring significant reduction in process failures, and improvements in efficiency, accuracy, and team morale. (Here is a short article about applying a Kaizen approach in an IT context.)
So, how and where can we improve our data management?
Where should you look to start identifying small improvements that might be implemented?
Consider the areas below with your team. Most likely you will find that you are very strong in some areas, but perhaps there are areas that have not been addressed at all. Start with the lowest hanging fruit, and bit by bit you will find that you are slowly filling the gaps and addressing the technical debt that every established data team faces.
First, analyze your data structures.
The applications, tools, and data processes in place for your company will impact the data structure that needs to be in place for it to be usable. Unfortunately, these requirements rarely align.
When you think about the flow of your data, think about consistency of format and type. As data flows into your system, it is often riddled with discrepancies in format, data type, and even the information it contains (but we will save that for another post).
As your data flows downstream toward the consumer, it should become more and more aligned in these areas. Why? Because the more points of contact that your technical teams must have with it (to transform it for particular use cases, etc.), the more points of failure you can have.
Strategic policies and governance and centralized data management can really help, but you don’t need an operational overhaul to improve!
In line with a Kaizen approach, try encouraging small changes in these areas:
Establish data standards
This will be an ongoing process. You will want to give thought to what your core standards should be, especially for mission critical data elements like identifiers, account numbers, etc., since these are more difficult to change once processes are mature. However, your standards will expand and refine as your business matures its data processes.
Adopt an enterprise modeling tool
Document and catalog your data standards using a modeling tool. Include all the metadata associated with your data objects and their relationships. The business will use the resulting documentation at every level (system administration, development, business analysis, and report consumption) for understanding and interpreting the data.
Transform your data with consistency
Wherever your transformation layer lives in your processes (and hopefully there are as few of these as possible), always architect toward your established data standards.
Establishing governance and centralized management can really help here, but feel free to start small! Apply these principles to new processes and only to established processes as they require other changes. Encourage a culture that celebrates these improvements and looks for opportunities to make things better.
Implement database source control
That’s right – employ a source control process for your database objects. Many companies do not take this step. However, having source control in place does not only protect your team from losing important data objects. It can also help ensure that new structures follow established standards when code reviews, pull request approvals, and other best practices are in place.
Structure your deployment process
Lastly, establish protocols around deployment. Some options include:
- Creating a deployment cadence that uses established deployment windows
- Setting up a change advisory board for reviewing changes before approving them to be deployed to production
- Designating deployment managers that are responsible for deploying code
- And, of course, you can always automate your deployments! Just be careful to include the appropriate guardrails.
Remember – slow and steady wins the race with continuous improvement.
Second, evaluate your data pipelines.
Outside of data structure, there are other data process considerations that need to be evaluated as well.
Accuracy & Reliability
- Are your data ingestion and replication processes accurate and reliable?
Sometimes when evaluating our pipelines, we find that issues with error handling, purge processes, SFTP, APIs, replication, logging or any number of other processes are causing duplicative, inaccurate, incomplete, or undelivered data transfers. Look out for these and correct them as you find them.
Maintenance & Scalability
Also, ask yourself these questions:
- Are you frequently stretching the limits of any of your allocated hardware, VMs, databases, or network resources?
- Are any of your system resources in need of upgrades or patching? Are you missing protocols to ensure that these are completed?
- Are there other systems, applications, technologies, or vendors that might suit your current or projected needs better?
- Are your data processes too slow? Do they struggle with the amount of data that must be processed by them?
If your answer to any of these questions is “yes”, then you have opportunities for improvement (slow and steady…).
Third, never forget about security with data management.
Security should always be top of mind when considering your company data. Here are some areas to evaluate.
Security – External
Review the security around the infrastructure supporting your company’s data processes for points of external connection. Pay particular attention to any processes that utilize third party tools or that export or extract data to/from external sources.
Security – Internal
For internal sharing and usage, security measures should be concerned with careful provisioning of access to data and systems. For lower-level systems, be sure to mask or de-identify any sensitive data.
Further, for sensitive or confidential data, give careful consideration to protecting against any intentional or unintentional data leaks. Areas to consider creating policies around include:
- Unsecured physical devices or paperwork
- Keeping only what data is necessary
- Emailing sensitive data
- Downloading data to personal devices
- What to do if a suspected data breach has occurred
Is your head spinning? Don’t worry!
Remember that data management is an ongoing process of continuous improvement, and we will delve into many of these topics more deeply in upcoming posts.
In the meantime, if you have a pressing need and could use some help detailing a roadmap, let us know! We love to help empower continuous improvement with our clients.
“We cannot become what we want to be by remaining what we are.”
– Max DePree