In the last post, we talked about applying the Kaizen approach to data management to achieve a culture of continuous improvement on our data teams.

In this post, we will use the KonMari method of simplification, recently made famous by Marie Kondo, as a lens for considering what to keep and what to purge in our business data repositories.

As always, we will also give some best practices for how to establish policies around archiving and purging your company data.

Data Lifecycle Management: Archiving and Deletion

As mentioned in previous posts, the final phases of Data Lifecycle Management are Archiving and Deletion.

These phases help to ensure that we keep and maintain only that data which is required for our business. But how do we determine what to keep active, what to archive, and what to purge?

We can apply some concepts from the KonMari simplification method to our data strategy here to help us decide.

Rule 1: Make the Commitment

Kondo states that the first step in KonMari is committing to achieving your goal. This may seem like an obvious first step for any endeavor, but many companies fail to establish a data retention strategy.

It is not uncommon for businesses to take the approach that keeping all data (sometimes even in a “hot”, or readily accessible, repository) is the way to go. This tactic usually stems from either:

1. an explicit belief that you cannot go wrong with keeping too much historical data
   or
2. from having no capacity to prioritize a retention strategy.

Either way, the “keep everything” strategy is misguided for 3 reasons.

First, the more data you keep, the more time it will take to recover in the event of a crisis.

Crisis can take the form of:

• a lawsuit or an audit that requires retrieval of specific information

or

• a natural disaster, human error, criminal activity or another event that demands restoration of data to a particular point in time.

In any case, restoration time is critical during these events. The more time it takes to retrieve the required data from your archive, the longer it will take for the business to recover.

Second, while data retention is a necessity, it is also a liability and entails responsibility.

Businesses must take the responsibility to respect consumer privacy rights very seriously. Part of this responsibility entails keeping consumer data for no longer than is required or for any purpose other than that for which the consumer gave consent. Even if your company does not fall under the regulatory jurisdiction of privacy laws like GDPR, CCPA, or PIPEDA, the business is nevertheless liable for securely and responsibly maintaining its consumer data.

With any data that is retained comes the possibility that it could be stolen, leaked, or misused. This risk is unavoidable, but preserving unnecessary archives of historical data is a liability that ought to be avoided.

Third, “Data stores don’t grow on trees…”

A well-crafted data strategy can reduce the financial cost of maintaining your data repositories, but increasingly large data stores cost the business money nevertheless.

There are also performance, time, system resource, and opportunity costs associated with maintaining large data stores.

So, in short – make the commitment to tidy up your unwieldy data repositories!

Rule 2: Imagine Your Ideal

Compliance regulations do some of the work of envisioning the ideal for us in the data world. Still, take the time to consider the ideal composition of your data repositories. Doing this can help you to think strategically about what should be kept, how and where to keep it, and for how long.

As mentioned in the previous post in this series, consider both regulatory requirements and the needs of the business for reporting, analytics, and strategic planning when determining what to keep. Consult business leaders and business analysts about what data is needed and for how long. You can even create a formal data contract for critical data elements in your business.

Rule 3: Finish Discarding First

Obviously, deleting data should always be done with extreme caution and forethought.

Nevertheless, once you have performed an audit of your data repositories and have determined your retention strategy, you should begin implementation by purging unnecessary data. We will discuss more about how to perform this action safely and according to best practices below.  

Rules 4 and 5: Progress by Category and in the Right Order

For the purposes of purging and archiving data, we should be thinking in a criticality/age matrix like the one below – beginning in the upper left corner and working down and to the right.

You should make incremental passes across departments in these stages, beginning with the oldest and least important data in each departmental area.

Rule 6: “Does it Spark Joy?”

Ok, ok – I admit this iconic final question from Kondo is much less suitable to data retention. However, there still may be an important (if less-than-perfectly-measurable) question that should be asked by data teams before coding a delete.

Does a business leader strongly prefer to retain certain data despite the lack of any clear regulatory or business-driven reason for doing so?

If so, keep it…unless doing so presents a serious risk or concern. If you feel there is a serious risk, continue to voice your concerns. Otherwise, just wait and continue to get clarification.

Now for some best practices…

Keep in mind that the practices listed here do not include the critical practice of backing up your production data, which has been discussed in previous posts (here, here, and here). Always make sure that backups are in place before beginning to archive or delete data.

1.     Replicate and archive data “in-flight”.

Archiving and/or replicating your data at various points in your pipeline is a best practice. Process failures in data pipelines are not uncommon, and you need to be able to recover data from earlier stages of the pipeline if you need to reprocess the data.

3 common examples of this practice, which should include purging data after an established retention period, are:

• Moving imported files to an archive folder
• Replicating transactional databases to a staging database before further processing by downstream systems
• Staging imported API data in its own table or database before integrating with internal systems.

2.     Archive in cold storage and protect the archive.

Consider how you will store data that has served its immediate purpose and has been determined to be a candidate for long-term storage. There are pros and cons to each method, and a combination of archiving methods may be appropriate for your different data sets. Here are some options and considerations.

• Onsite physical storage
  • Pros: ease of access in an emergency, familiar technology
  • Cons: vulnerable to tampering, theft, and physical damage/natural disasters
• Offsite storage (tape, optical disk, magnetic hard drives)
  • Pros: well-established, relatively inexpensive, usually more secure than onsite
  • Cons: slower recovery times
• Cold cloud storage
  • Pros: speed and ease of recovery, alleviated burden of maintenance, built-in security, inexpensive
  • Cons: potentially less familiar to established IT departments than traditional methods
• Data lake
  • Pros: accessibility, speed and ease of recovery, suitable for use with emerging technologies, built-in security
  • Cons: less-established, steeper learning curve, potentially expensive, requires careful governance of user access

3.     Don’t forget critical data that is managed by third parties.

Over time, businesses may find that a significant body of company data resides in data stores managed by third parties. Since maintenance has often been delegated to vendors in these cases, archiving this data is sometimes overlooked.

While there may be an ability to set a retention policy, it can be very beneficial for many reasons to work with your vendors to set up an extraction process to archive a copy of your data from these systems into your own repositories as well.

4.     Establish retention schedules and procedures for purging data.

Business, regulatory, and legislative needs dictate what should be saved and for how long, and these may differ between data sets. Establishment of policies and procedures for deleting data will ultimately be the responsibility of data owners.

These policies should address the following areas:

• Who is authorized to purge data?
• In what maintenance windows can this process occur since deletion can go slowly and data processing jobs/replication must be disabled?
• How will notification be given to the business?
• What validation and integrity checks must be in place?
• What rollback procedures will be used if necessary?

Just the tip of the iceberg…

There is much, much more to say about all these topics.

If you have made it as far as committing to cleaning up your data but the rest seems overwhelming, never fear! There are many vendors that are happy to help with all levels of assistance.

If you have a good handle on your archiving and deletion processes but would like assistance with a SQL Server implementation of them, reach out! We are here to help.

The post Archiving and Deletion Strategy…KonMari for Data Management? appeared first on The SERO Group.

This refrain is more common than you think in IT departments of all sizes. Or maybe you live that reality every day and are fully aware that clunky, error-laden processes eat away at your team’s efficiency (and morale).

Data management and continuous improvement may sound like they should always go together, but they often don’t.

Many times our data management practices involve too many business-critical data processes that break regularly and need to be improved, but we have no time to make the needed improvements because there are so many data processes that need to be “managed” (a.k.a. remediated regularly).

So, how can we break this cycle?

Welcome back to the second of three posts on how to refine your strategy for Data Lifecycle Management (DLM)!

In this post, we will focus on Data Management as the second of the three DLM stages: Data Collection, Data Management, and Data Deletion.

Kaizen for Data Management

The Kaizen approach, famously championed by the Toyota corporation, suggests that small organizational changes can lead to a culture of continuous improvement. This culture will ultimately lead to better processes, greater efficiency, improved outcomes, and increased morale.

The Kaizen Institute states,

“As part of the corporate culture, continuous improvement becomes an ongoing process integrated into the organization’s daily activities. Employees are encouraged to challenge the status quo, suggest ideas, and implement improvements. Continuous learning and development are valued, and mistakes are seen as growth opportunities.”

This means that adopting a Kaizen approach to your data management strategy can be a lever for driving a continuous improvement culture on your data team without sweeping, drastic changes.

Small improvements to existing processes can slowly bring significant reduction in process failures, and improvements in efficiency, accuracy, and team morale. (Here is a short article about applying a Kaizen approach in an IT context.)

So, how and where can we improve our data management?

Where should you look to start identifying small improvements that might be implemented?

Consider the areas below with your team. Most likely you will find that you are very strong in some areas, but perhaps there are areas that have not been addressed at all. Start with the lowest hanging fruit, and bit by bit you will find that you are slowly filling the gaps and addressing the technical debt that every established data team faces.  

First, analyze your data structures.

The applications, tools, and data processes in place for your company will impact the data structure that needs to be in place for it to be usable. Unfortunately, these requirements rarely align.

When you think about the flow of your data, think about consistency of format and type. As data flows into your system, it is often riddled with discrepancies in format, data type, and even the information it contains (but we will save that for another post).

As your data flows downstream toward the consumer, it should become more and more aligned in these areas. Why? Because the more points of contact that your technical teams must have with it (to transform it for particular use cases, etc.), the more points of failure you can have.

Strategic policies and governance and centralized data management can really help, but you don’t need an operational overhaul to improve!

In line with a Kaizen approach, try encouraging small changes in these areas:

Establish data standards

This will be an ongoing process. You will want to give thought to what your core standards should be, especially for mission critical data elements like identifiers, account numbers, etc., since these are more difficult to change once processes are mature. However, your standards will expand and refine as your business matures its data processes.

Adopt an enterprise modeling tool

Document and catalog your data standards using a modeling tool. Include all the metadata associated with your data objects and their relationships. The business will use the resulting documentation at every level (system administration, development, business analysis, and report consumption) for understanding and interpreting the data.

Transform your data with consistency

Wherever your transformation layer lives in your processes (and hopefully there are as few of these as possible), always architect toward your established data standards.

Establishing governance and centralized management can really help here, but feel free to start small! Apply these principles to new processes and only to established processes as they require other changes. Encourage a culture that celebrates these improvements and looks for opportunities to make things better.

Implement database source control

That’s right – employ a source control process for your database objects. Many companies do not take this step. However, having source control in place does not only protect your team from losing important data objects. It can also help ensure that new structures follow established standards when code reviews, pull request approvals, and other best practices are in place.

Structure your deployment process

Lastly, establish protocols around deployment. Some options include:

• Creating a deployment cadence that uses established deployment windows
• Setting up a change advisory board for reviewing changes before approving them to be deployed to production
• Designating deployment managers that are responsible for deploying code
• And, of course, you can always automate your deployments! Just be careful to include the appropriate guardrails.

Remember – slow and steady wins the race with continuous improvement.

Second, evaluate your data pipelines.

Outside of data structure, there are other data process considerations that need to be evaluated as well.

Accuracy & Reliability

• Are your data ingestion and replication processes accurate and reliable?

Sometimes when evaluating our pipelines, we find that issues with error handling, purge processes, SFTP, APIs, replication, logging or any number of other processes are causing duplicative, inaccurate, incomplete, or undelivered data transfers. Look out for these and correct them as you find them.

Maintenance & Scalability

Also, ask yourself these questions:

• Are you frequently stretching the limits of any of your allocated hardware, VMs, databases, or network resources?
• Are any of your system resources in need of upgrades or patching? Are you missing protocols to ensure that these are completed?
• Are there other systems, applications, technologies, or vendors that might suit your current or projected needs better?
• Are your data processes too slow? Do they struggle with the amount of data that must be processed by them?

If your answer to any of these questions is “yes”, then you have opportunities for improvement (slow and steady…).

Third, never forget about security with data management.

Security should always be top of mind when considering your company data. Here are some areas to evaluate.

Security – External

Review the security around the infrastructure supporting your company’s data processes for points of external connection. Pay particular attention to any processes that utilize third party tools or that export or extract data to/from external sources.

Security – Internal   

For internal sharing and usage, security measures should be concerned with careful provisioning of access to data and systems. For lower-level systems, be sure to mask or de-identify any sensitive data.

Further, for sensitive or confidential data, give careful consideration to protecting against any intentional or unintentional data leaks. Areas to consider creating policies around include:

• Unsecured physical devices or paperwork
• Keeping only what data is necessary
• Emailing sensitive data
• Downloading data to personal devices
• What to do if a suspected data breach has occurred

Is your head spinning? Don’t worry!

Remember that data management is an ongoing process of continuous improvement, and we will delve into many of these topics more deeply in upcoming posts.

In the meantime, if you have a pressing need and could use some help detailing a roadmap, let us know! We love to help empower continuous improvement with our clients.

“We cannot become what we want to be by remaining what we are.”

– Max DePree

The post Data Management Strategy: A Kaizen Approach appeared first on The SERO Group.