The answer is not simple.

Data roles are diverse and constantly evolving. Similarly, the lines that separate data disciplines are inherently blurred. The reality faced by active data professionals is also complex, with business and project requirements often requiring them to extend their expertise across disciplines. Consequently, data professionals often wear many hats.

Still, it’s useful for those entering the field or looking to hire a data professional to understand some of the important distinctions between data disciplines.

Here are just a few.

Common Data Roles

Data Architect

Data architects design the overall blueprint for your organization’s data environment. They define how data is stored, organized, integrated, and accessed across systems. They also ensure that your data infrastructure is scalable, secure, architected for efficient retrieval, and aligns with your long-term business goals.

Database Administrator (DBA)

DBAs are the administrators of your database systems. They possess a deep knowledge of the database engine itself, including all its native functionality and features. They are also responsible for keeping databases updated, backed up, secure, and performing optimally. DBAs also manage database upgrades and migrations, as well as database recovery in a disaster or emergency.

Data Engineer

Think of data engineers as the builders of your data infrastructure. They design, construct, and maintain the pipelines that collect, store, and process your data. Their toolkit is diverse and often includes programming languages like Python and SQL. It is also common to leverage cloud platforms like Azure, AWS, or GCP. These professionals ensure that your data is accessible, reliable, and ready for analysis.

Data Analyst / Business Intelligence Analyst

Data analysts are the storytellers of the data world. They take the data that engineers have prepared and draw out insights through creating reports, dashboards, and visualizations. Furthermore, their strong analytical skills and ability to recognize patterns allow them to turn data into knowledge. They tell stories with data to help guide decisions. Tools often include Excel and BI reporting platforms like Tableau or Power BI.

Data Scientist

Data scientists take analytics to a deeper level. They use advanced statistical techniques and machine learning to uncover hidden patterns that are often difficult for humans to detect. Similarly, the algorithms used by data scientists can predict future trends, and the outputs of their models are used to drive decision-making. Data scientists possess mathematical expertise, programming skills in languages like Python or R, and domain knowledge relevant to their industry.

Machine Learning Engineer

Machine learning engineers take the models created by data scientists and make them operational in real-world production environments. They leverage both data science and software engineering skills. ML engineers are responsible for building the systems that deploy, monitor, and scale data science models. They also manage these systems to ensure that they deliver accurate and timely predictions once deployed in a real-world context.

Tips for Choosing Your Future Data Role

Shadowing some different data professionals is a great way to get started. Depending on which role appeals to you, there are different pathways to getting started.

If you have a background in system administration, data architecture and database administration may be good avenues to investigate. Likewise, people who have enjoyed building their own home labs may also enjoy these data roles.

Similarly, if you love automation and problem-solving for technical efficiency, one of the engineering roles may be right for you. Engineers enjoy designing and building solutions for technical challenges.

Finally, if you have a mind for analytics and statistics, an analyst or data science role may be a great fit. These roles uncover the root causes of the problems under investigation in order to unlock potential solutions.

There are a variety of ways to kick-start your journey for any of these data disciplines.

1. Self-Education. There are many resources available online that can guide you through learning specialized skills. Set up a home lab environment in which to safely practice. Start to build a profile of projects and/or certifications that can showcase your new skills.
2. Bootcamps. The number and variety of bootcamps for data have increased dramatically in the last 5 years. If your schedule allows for participation in one of these intense programs, they can be a great way to upskill rapidly.
3. Formal education. If you are looking to go into a highly specialized role in a particular knowledge domain or industry, obtaining an advanced degree can be a great way to get started.

Determining which path to take will depend on what appeals to you and on the circumstances in which you are beginning your journey. The key, however, is just taking the first step.

Tips for Hiring: Which Data Role Do You Need?

Just as the path to becoming a data professional depends on individual circumstances, the type of data professional to hire will also depend on your organization’s needs and where you are in your data maturity journey.

Here are some hiring considerations for several common scenarios:

Just Starting Out

If you are building your data capabilities from the ground up, a data engineer with versatile skills is a great first hire. While partnering with system administrators and business experts, as well as potentially seeking external assistance from a DBA or data architect, a data engineer can lay the foundation for your future data work.

Business Optimization / Improved Operations

If you are looking to use your data for improved operations, a data or business intelligence analyst can work with business partners to track key metrics, identify trends, and drive data-driven decisions to find ways to improve. Likewise, by developing an understanding of the business, this person can work as a bridge between business and IT teams to help harness the full potential of your data.

Complex or Outdated Data Environment

A data architect can bring order to chaos by ensuring that your data is well-organized, accessible, and scalable as your organization grows and expands. Data architects can also be instrumental in reshaping legacy structures to meet current business requirements and make the best use of modern technologies.

Predictive Insights and Automation

Data scientists and machine learning engineers are ideal for building models that can make important predictions, optimize processes, and/or automate complex AI tasks. It is important to note, however, that having a robust, well-governed data infrastructure is a prerequisite for success for these types of initiatives.

Diverse Needs, Large-Scale Projects, or Complex Remediation

Partnering with an external data team with diverse areas of expertise can be an affordable way to: remediate complex problems, make infrastructure improvements rapidly, or design and implement large-scale solutions. Leveraging this type of support also allows you to access professionals with expertise in different data roles and may offer flexibility for scaling up or scaling down your level of support as needed.

Want to work with The SERO Group?

Want to learn more about how SERO Group helps organizations take the guesswork out of managing their SQL estate? Schedule a no-obligation discovery call with us to get started.

The post What’s in a Job Title? Understanding Changing Data Roles appeared first on The SERO Group.

The risks of delaying code analysis

Delaying an analysis of code compatibility is surprisingly common for businesses undertaking a migration to the cloud. Early cost-benefit analyses often address the hardware and infrastructure changes involved but can sometimes neglect to consider the impact of required code changes.

Infrastructure concerns are critical considerations when evaluating a move to the cloud. However, limiting our analysis to these considerations may hide the costs and risks associated with any necessary re-engineering. Unfortunately, it is very possible for these hidden risks and costs to turn out to be deal-breakers. To avoid a sticky situation, learn about these factors up front before investing time and energy into migration preparations.

Azure options: SQL VM vs. Managed Instance vs. SQL Database

It is important to note that a “move to the cloud” can come in many forms. Some examples are migrations to a hybrid environment, “lift and shift” moves to Azure-hosted VMs or SQL Managed Instances (SQL MI), or full or partial migrations to multi-tenant Azure SQL Databases (SQL DB). You will need to know what type of migration is being considered before evaluating code changes since the different options have different levels of compatibility with SQL Server.

Here’s a quick breakdown of the options in Azure with SQL Server compatibility.

IaaS Option

SQL Server on an Azure VM
Since this option constitutes a full installation of SQL Server on a dedicated Azure-hosted virtual machine, there are no code or feature incompatibilities to be concerned about. Azure SQL VMs achieve complete feature parity with on-premises SQL environments. With this option, the primary difference between the Azure implementation and an on-premises installation is the management of the underlying server.

PaaS Options

Azure SQL Managed Instance
SQL MI boasts “near 100% compatibility” with the latest Enterprise Edition of the SQL Server database engine, while still including automated backups, patching, and high availability of the SQL environment. This option uses a single-tenant database engine intended to enable the least disruptive migration from an on-premises or Azure-hosted SQL Server instance to a full PaaS environment. This means that many of the incompatibilities that exist with Azure SQL Database are minimized or eliminated with Azure SQL MI. However, functionality that requires access to the file system or OS is still impacted.

Azure SQL Database
Azure SQL DB is another fully managed PaaS option consisting of a multi-tenant database engine that is optimized for cloud-native applications. While this option is generally less expensive than Azure SQL MI, there is less overlap with SQL Server and greater potential for code or data flow incompatibilities.

10 Common Incompatibilities

There are several very helpful tools (see the “Resources and tools” section below) that can help you identify data flow issues prior to migration. As you go through your analysis, keep in mind that Microsoft has established workarounds for a good number of these incompatibilities, so their presence in your code does not necessarily mean they are a barrier to migration.

For a high-level overview, here are some of the most common sticking points we see for migrating SQL code. The differences between SQL MI and SQL DB are included where applicable, as well as some potential workarounds.

1. Uses linked servers

Linked servers can be used in SQL MI to access SQL Server and Azure SQL Databases without distributed transactions. SQL DB requires the use of elastic queries instead.

2. Performs cross-database queries or transactions

These are supported with SQL MI, but not with SQL DB. In SQL DB, cross-database queries may be able to be converted to elastic queries.

3. Uses Database Mail

This is available for SQL MI but not SQL DB. There are workarounds available for sending email in the Azure platform, but they will require some re-engineering.

4. Uses system tables, views, functions, or stored procedures

Some system objects are available in both SQL MI and SQL DB but not all. Consult Microsoft’s documentation for a full comparison of what is available.

One important note is that the amount of space available to tempdb is provisioned in both SQL MI and SQL DB based on the number of cores available and the service tier licensed. Consult the documentation of each for details.

5. Accesses Windows command line or file system

Neither SQL MI nor SQL DB supports direct access to the file system or the Windows command line.

One workaround is to migrate files to Azure Blob Storage or Azure Files. For SQL MI, with the appropriate security and firewall configurations, it is also possible to establish connectivity between your Managed Instance’s VNet and the location of an on-premises file share.

SQL MI also supports SSISDB configuration and the Integration Services Catalog, allowing SSIS packages to be used for file manipulation. Azure Data Factory can also be leveraged to load and transform files for both SQL MI and SQL DB. An Azure-SSIS Integration Runtime (IR) can be installed and configured, and SSIS packages can be run directly from Azure Data Factory.

6. Uses change data capture (CDC)

Change data capture is supported for SQL MI. It is also supported for SQL DB, but only in the S3 service tier and above.

7. Uses BULK INSERT or OPENROWSET

BULK INSERT and OPENROWSET are only supported from a supported Azure file source (e.g.: Azure Blob Storage or Azure Files).

8. Uses .NET Framework: common language runtime (CLR)

CLR support is not available for SQL DB, but it is available in SQL MI with some important differences.

9. SQL Server Agent

SQL Server Agent is not available in SQL DB, and elastic jobs should be used instead. In SQL MI, SQL Server Agent is supported with important differences.

10. Uses semantic search

Full-text semantic search is not available in either SQL MI or SQL DB.

Other important things to remember when migrating to a PaaS environment

• High availability: Since high availability is included in the PaaS offerings, SQL Server functionality and syntax connected with Always On Availability Groups is not supported.
• Maintenance: Updates, patches, backups, and restores are likewise managed automatically in the PaaS offerings. Therefore, associated T-SQL syntax will not work in SQL DB and will be different for SQL MI.
• Credential management: Windows authentication is not supported in SQL DB, and is replaced by Windows Authentication for Microsoft Entra in SQL MI.
• Collation: Catalog collation is set when an instance (SQL MI) or a database (SQL DB) is created, and it cannot be changed afterwards.

Resources and tools

I hope this provided you with a jump-start for thinking about whether your SQL Server code is Azure-ready.

Here are a few more resources and tools that can help you take the next steps toward a full compatibility analysis:

• Data Migration Assistant – Microsoft’s robust tool for enabling database compatibility assessments, recommendations, and migration assistance.
• Azure Migrate – this service can be used as a start-to-finish hub for planning and facilitating a cloud migration.
• Pro Database Migration to Azure – An excellent and comprehensive book covering the best practices for successful on-premises migrations to the Azure cloud platform.

Want to learn more?

Want to learn more about how The SERO Group helps organizations prepare for a SQL Server cloud migration? Schedule a call and let’s talk.

The post Is your SQL Server Code Ready for Azure? appeared first on The SERO Group.

This interpretation misses a key benefit of data governance, however: competitive advantage. We cannot diminish the critical importance of regulatory compliance and its role in shaping governance efforts, but it’s important to understand that this is not the whole picture when it comes to data governance.

Governing Data as an Asset

Data is a business asset, not unlike cash. If you asked any business leader whether their cash flow processes needed controls to ensure their accuracy, integrity, and protection, they would likely give you a quizzical look and assume you were baiting them into some trap…of course their cash flow processes need controls.

Similarly, data has the potential to guide and inform nearly every aspect of a business. To leverage data effectively, businesses must ensure its integrity in an ongoing and reliable way. From financial reporting to performance evaluation to operational efficiency, data quality sits at the heart of modern businesses. Later, we will see some examples of how data quality and integrity issues can have significant impacts on SMBs.

Additionally, company data assets need to be secure and protected—arguably even more so than other company assets. This is because many personal data elements do not ultimately belong to a business. These considerations impact regulatory compliance and are essential for effectively and responsibly leveraging data as the valuable asset that it is.

The policies and processes that ensure the availability, usability, integrity, and security of data assets are what we mean by data governance.

Did you know that…

• 97% of data leaders state that their companies have experienced the costs of disregarding data quality and integrity in the form of lost revenue opportunities, inaccurate performance forecasting, and/or poor investments.
• More than 87% of small and mid-sized businesses (SMBs) collect or process sensitive customer data that could be compromised.
• Small businesses spend an average of $955,429 to restore normal business operations in the wake of successful attacks.

AND…

• About 85% of data science and analytics projects fail due in part to disregarding data governance processes.

Four Common Challenges for SMBs

Challenge #1: No Buy-In

To effect change within an organization, you must secure buy-in from key constituents. For data governance, these important players begin with executive leadership and extend through the organizational hierarchy to front-line staff. However, some SMBs question whether data governance has any real applicability for them. In response, consider: Do you store sensitive customer, client, or patient data? Do you keep sales, productivity, performance, or program data for decision-making? If you answered yes to any of those questions, data governance principles do have applicability in your environment.

Business leaders and decision-makers should prioritize data governance because their decisions directly impact the company’s value and profitability. Without data governance policies, decision-makers can only hope that their data-driven decisions are well-founded. Inaccurate data can lead to poor decisions in ways that are difficult to detect or explain even after the fact, leaving decision-makers accountable for the outcome.

Securing employee buy-in is equally important when implementing data governance. Buy-in should stem from an understanding of the shared responsibility for the data that drives business outcomes. A collective sense of accountability for data quality works to prevent resentment and misunderstandings towards your policies that could undermine even the best data governance efforts.

Tips for Success:

• Elicit interest and assistance from vested stakeholders. Leverage opportunities to ease business pain points through data governance. I’ll list some examples in “Data Governance in the Wild” below.
• Pave a step-by-step path to mature your data culture. Start with education, delegated responsibility, and distributed data ownership.
• Align policies with the existing strategic objectives of the company. Be specific about the expected business outcomes of your efforts.
• Empower business stakeholders to take ownership of and make decisions about business data.
• Create strategic goals and an initial roadmap. This will work best if it is an ongoing organizational effort that is largely led by the business, not by IT.    

Challenge #2: No Budget

Chances are that unless improving data governance is already a top business priority, perhaps because of a recent security incident, audit, or a particularly problematic data irregularity, you will be up against budget constraints when introducing formalized data governance to an SMB.

This is a challenge, but not a barrier. Yes, there are tools on the market that can be very helpful, but they are by no means necessary for SMBs—and certainly not at the outset.

Tips for Success:

• Start small and go slow. Introduce change gradually by beginning with what you can accomplish internally. Define and document business logic for your most critical data objects, and then introduce business processes to enforce that logic.
• Treat data governance as a process of continuous improvement rather than a costly one-time project. Small, inexpensive wins can produce significant results.
• Avoid a box-checking mentality. Use internal data audits to showcase regulatory compliance, but don’t let minimal standards dictate strategy. Prioritize business advantage over compliance, focusing first on areas with immediate value, such as labor cost savings, enhanced reporting, or boosted analytical capabilities.
• Leverage tools strategically. Develop your strategy and framework first. Look to tools second, and only for ease of implementation and execution. Tools alone will not provide true data governance.

Challenge #3: No Champion

You know that data governance is important for your organization and that someone needs to advocate for it. You also might not have the authority, capacity, or desire to be that person. In that case, use the suggestions about securing buy-in from Challenge #1 to find or create a data governance champion within your organization.

Tips for Success:

• Business and IT need to collaborate for effective data governance. There should be a designated senior IT team leader who is collaborating closely with at least one senior business leader. In a small organization, this could simply mean your one IT resource working with the business owner. Regardless of the size of the organization, the effort should not be one-sided.
• Champions should not limit their scope to policies alone. Having centralized documentation, tools, and processes at the ready will help to avoid individuals and teams developing their own tools and processes to implement the policies their own way. Disparate processes can lead to inconsistency and diminished success.
• Champions need to be committed and persistent. Don’t roll out data governance processes and forget about them. Remember that education and training will need to be ongoing, as will the refinement of governance practices to keep pace with technology and changing business requirements.
• Champions should consider the needs of the employees that work with the data when creating policies and approving tools. If your governance initiatives are seen as an unnecessary burden, workarounds will be developed that could compromise your efforts.

Challenge #4: Internal Skills Gap

You have the needed organizational buy-in, a small budget, and even a champion, but your IT team is small (or outsourced), without any data professionals on staff. Where can you begin?

Remember, the key is to go slowly and implement organically. The full implementation of data governance will ultimately require technology expertise, but defining rules, requirements, and workflows may not. Start there, then reach out for help with technical implementation when you’re ready.

Tips for Success:

• Research any compliance regulations applicable to your business sector. Perform an analysis of how your business is meeting these requirements. Where you find gaps, assess a strategy for how to fill those gaps. Implementation can be left to a third party as needed, but be sure that you understand what needs to be done. If you are unsure about how to get started, use some of that budget to reach out for support.
• Perform a security audit of your data systems. Adjust and delete credentials and permissions as appropriate. Review third-party system access. Inventory inbound, outbound, and internal data pipelines and evaluate for best practices. Then set up ongoing processes and/or reports to assist with security monitoring.
• Look for critical data objects that need to be formally defined. Beginning with mission-critical data elements, document all business logic and metadata requirements associated with these elements in a data dictionary or catalog. This doesn’t need to be elaborate, but it should cover all key data elements. Start in one business area and expand out from there.
• Assign ownership and accountability. Look for data owners who are invested in the quality of the data within their domain because of their connection to the processes that produce or consume the data.
• Upskill and/or seek third-party guidance as required. Provide access to resources for interested staff to get trained to become data owners, stewards, and partners in data governance. Reach out to third parties for guidance in areas that are outside the internal expertise of the organization.

Data Governance Challenges in the Wild

Below are some real challenges faced by SMBs that could be solved with improved data governance processes.

Local Restaurant Chain

A small, local restaurant chain encountered a discrepancy between a company report on drink sales and individual location reports generated in Excel. This irregularity surfaced after a competition to incentivize staff to sell more drinks by holding a competition between locations with a cash prize. At the end of the competition, when managers compared their individual reports, Store A had sold the most drinks. When using the central company report, Store B had. Which was correct?

Upon investigation, one of the restaurant locations had long ago coded milkshakes as drinks. All of the other locations coded them as desserts. This was never audited or corrected. As a result, the individual reports that were rolled up by individual menu item codes produced a different result than the central report that was rolled up by the “Drink” category. This confusion affected competition results as well as drink and dessert sales analytics and other location-to-location comparisons.

Small Food Manufacturer

A large grocery store chain approached a regional manufacturer of salad dressings and condiments, requesting the production of one of its products as a generic for the chain. Similarly, a bulk food retailer asked the same manufacturer to produce a bulk version of the same product. The contracts were accepted, but the original product, the generic version, and the bulk version of the product are all assigned different product codes without being rolled up to any parent product category. When the accounting office ran their standard report of overall sales trends by product, they had no way of recognizing that these three items represented the same underlying product and ended up producing a skewed and incomplete product analysis.

International Boutique Wholesaler

A private international wholesaler with dispersed brick-and-mortar locations was having continual problems with their inventory. Customers complained that the website frequently showed products as available when they were not. Retail associates complained that the inventory in the POS was incorrect and frequently showed negative inventory when stock was actually present in the stores.

One of the issues in this scenario was that individual store inventory was manually counted and entered with different workflows, lacking a centralized mechanism for tracking inventory. Furthermore, associates often used the “miscellaneous” category to sell products that were at the stores but didn’t have a corresponding inventory item recognized in the system. These issues produced a myriad of operational, financial, and analytic discrepancies, as well as increased costs in the form of labor inefficiencies, lost revenue, and misplaced or stolen inventory.

Social Services Organization

A social services organization gathers information about its clients to produce an annual statistical analysis of clients and outcomes. This nationally distributed, highly regarded report serves as a source for scholarly research and influences funding for numerous public and private institutions.

However, the organization faces challenges in bringing consistency to its intake processes. Since many clients arrive in crisis, intake steps that can be skipped by staff often are because intake personnel are in a rush and do not realize the importance of each step. Furthermore, the application being used to collect this valuable information does not require answers to many questions that are deemed critical for this organization and the annual report. The organization has attempted many process changes to improve the consistency of this workflow. However, the problem still persists and is consistently cited as a limitation of their statistical findings.

Is data governance important for SMBs?

This is just a small handful of examples of how a lack of data governance can affect SMBs. In fact, once you’re on the lookout for data governance issues, you may start to notice them everywhere.

Additionally, we have only begun to touch on the security and compliance aspects of data governance. Many of the challenges to implementation outlined here are less prominent when a regulatory requirement is at play since compliance is mandated. Even when regulations play a less obvious role, most of us are aware of the gravity of security risks. It is important to note, however, that this security risk is even greater for SMBs than for the largest companies. According to Veeam’s 2023 Data Protection Trends Report, 85% of ransomware attacks targeted small businesses.

The take-away? If your business or organization retains data, and very few (if any) businesses do not, formalizing some level of data governance will help you to securely and responsibly leverage that asset to drive your business.

Further Reading

• • More interesting statistics

• • A privacy audit checklist

• • A guide to data governance for small businesses

• • Data governance frameworks

• • Amazon’s guide to data governance for SMBs

Want to continue the conversation about data governance for SMBs?

To learn more, you can click here to access a recording of our May 14, 2024 webinar on Data Governance for SMBs.

Need help? We can help you tailor practical data governance solutions to meet the specific needs of your SMB. Schedule a call or send us an email. 

The post Data Governance in Action: 4 Challenges for Small and Mid-Sized Businesses (SMBs) appeared first on The SERO Group.

Here are 7 important tips for managing your transaction logs:

1. Select the optimal recovery model and backup strategy for your database.
2. Configure Auto-Growth setting for optimal performance.
3. Configure the Max File Size of the transaction log.
4. Store log files on a separate drive from data files.
5. Minimize the use of log shrinking.
6. Use only one log file.
7. Monitor log file size and growth regularly.

Now, let’s break these down one at time.

Recovery Model and Backups

First, the #1 factor to consider in managing SQL Server transaction logs is the recovery model settings of your databases. Additionally, you must consider the correlated backup processes required for recovery for each model.

In a recent post, we explained that if a database is in Simple Recovery mode, SQL Server will take full or differential backups as scheduled. In this mode, however, the database can only be restored to the point of the last backup. Transaction logs are flushed automatically by SQL Server as data is committed, so there is no log maintenance or transaction log backup required. This is a good strategy for non-critical databases where your recovery point objective can be the point of the last full or differential backup.

By contrast, databases in Full or Bulk-Logged Recovery mode rely on the DBA to maintain the logs. Administrators must ensure that they are truncated frequently enough by a transaction log backup to prevent them from filling up. A common practice used by many DBAs is to use Ola Hallengren’s scripts for establishing the backup chain. Additionally, a common cadence is to perform weekly full backups, daily differential backups, and transaction log backups every 15 minutes.

Auto-Growth Setting

The Auto-Growth setting tells SQL Server how much to grow the log file each time that it needs to expand. The size of this growth should not be too small, since log file growth operations are slow and can impact query performance. This impact is due to the fact that log files have been historically unable to make use of the instant file initialization (IFI) option.

Notably, there have been some improvements in this area if you are on SQL Server 2022. Specifically, fewer VLFs are created internally by SQL Server logging processes. Additionally, the transaction log can make use of IFI if it is enabled under the normal requirements and auto-growth is not set to more than 64 MB (the default setting for 2022). (More below*)

For earlier editions, however, transaction logs cannot use IFI. Therefore, using a larger auto-growth setting of 256 MB to 1024 MB (or more) can be beneficial for performance.

Additionally, setting the initial size of the log file to 20-30% of the initial data file size is a good practice. Pay particular attention to the initial size for the tempdb log file. Tempdb will always be shrunk to its initial size after a restart and will have to re-grow to a standard size. So, you do not want this setting to be unreasonably small to start.

How to Adjust in SSMS

Right click on the database and navigate to the properties tab. Click on Files in the list on the left.

Then click on the ellipsis next to the log file,

and change the File Growth to the appropriate value.

You can also run the below query to update the setting:

ALTER DATABASE [YourDB] MODIFY FILE (name = 'YourDB', FILEGROWTH=256MB);

Max File Size Setting

Configuring the Max File Size setting is not necessary for all databases. However, doing so can be useful for databases where there is a reasonable likelihood that there could be a runaway query or process that could cause the log to grow to the point that it consumes all space on the server and causes SQL Server to fail.

If this setting is in place in such an event, SQL Server will be able to read but unable to commit transactions and will throw errors. It will not, however, fill the entire server and cause SQL Server to fail.

How to Adjust in SSMS

Follow the same directions as above, and in the Autogrowth window, change the Maximum File Size:

Or use this query:

ALTER DATABASE [YourDB] MODIFY FILE (name = 'YourDB_logfilename', SIZE=64000MB);

Log Storage

Best practice is to isolate your log files on a separate physical drive from your data files for optimal performance. This is to prevent the intensive sequential write workload required for the logs from interfering with the random write workload used for the data files.

Log Shrinking

Occasionally, we will see databases with log shrinking employed as a part of routine maintenance or the Auto-Shrink feature turned on. Both should be avoided.

Log shrinking should be reserved for occasions when there is a large amount of unused log space or when bulk operations may have caused the log to grow more than it ordinarily would. (Note that databases that see many bulk operations may benefit from the Bulk-Logged recovery model setting, which suspends logging during qualifying bulk operations.)

Using Auto-Shrink should only be done with caution and full awareness of the implications, since it can lead to significant performance issues. Microsoft’s documentation does a good job of outlining the feature and its implications.

How to Identify and Shrink Appropriate Logs in SSMS

Obtain information about log space consumption using:
DBCC SQLPERF(logspace)

If necessary, files with an unusually large percentage of unused log space can be shrunk using:

DBCC shrinkfile(YourDB_log, 1024) -- shrink log to 1 GB

The appropriate size to which to shrink the log will depend on the database. Shrinking to an average log size makes sense. (See below for information about obtaining this.**)

Use One Log File

Only set up multiple log files for a database in unique situations where space is limited (like in the case of a full log drive). If this is required, you should remedy this situation as soon as possible.

Understand that setting up multiple log files does not allow SQL Server to write to them in parallel and does not improve performance. In fact, having multiple large log files will significantly slow down your recovery if you need to restore your database in a DR scenario.

Monitor Regularly

Once you have your databases configured according to best practices and your maintenance processes established, your transaction logs could easily operate without issue for years!

It is still good practice, however, to monitor the log file size, growth, and performance for issues before they have a chance to escalate to crisis events.

There are multiple tools and services on the market that can assist with this, including our own SEROShield, but you can also monitor with reports that provide insight through queries like the following:

Monitoring Log Usage

DBCC SQLPERF(logspace)

SELECT * FROM sys.dm_db_log_space_usage

Why a Log is Awaiting Truncation

SELECT [name] as DatabaseName, [log_reuse_wait_desc] FROM sys.databases

Monitoring Virtual Log Files (VLFs)

DBCC LOGINFO

Want to learn more?

Here are a few more resources about some of the topics we touched on:

• *SQL Server 2022 Changes – a very interesting demo of the new functionality
• Instant File Initialization (IFI) and the Transaction Log – Microsoft documentation
• More Information and Best Practices
• Transaction Log Growth Triage – a nice post on triaging a transaction log growth event
• **Tracking Log Usage – a process for tracking log usage over time
• More on Monitoring
• Virtual Log Files (VLFs)

Your SQL log file configuration can affect performance. It can also affect the recoverability of key databases. If you’re concerned about your log file configuration, we’re happy to help. Contact us to set up a short call to discuss.

The post Log-Wrangling 101: 7 Tips for Managing Your SQL Server Transaction Logs appeared first on The SERO Group.

First, what is the transaction log?

The SQL Server transaction log is a crucial element within each database that records all database changes across time. This log enables point-in-time recoverability and rollbacks in the event of mistakes, corruption, or failure. This recoverability is critical to ensuring the integrity and resilience of the database.

What is contained in the transaction log?

There are several key concepts and log components to be aware of when dealing with SQL Server’s transaction logs.

1.     Log Records

Within SQL Server’s transaction logs, individual change operations are enclosed within log records. Each log record represents one UPDATE, INSERT, or DELETE operation in the database and is associated with one transaction. Each transaction, however, may have numerous log records.

Every data modification made in the database has one or more log records associated with it. Log records contain either the logical operation performed with the modification made, or they contain before and after images of the modified data. Start and end times of each transaction are also recorded.

2.     Log Blocks

A log block is the basic unit of I/O when it comes to transaction logs. The sizes of the log blocks vary but are at least 512 bytes and can be as large as 60 KB. Log blocks can contain multiple log records or none. According to Microsoft, “a log block is a container of log records that’s used as the basic unit of transaction logging when writing log records to disk.”

3.     Virtual Log Files (VLFs)

When considering the database transaction log, it is important to keep in mind the concept of the physical log file and its composite VLFs, or virtual log files.

The division of the log into multiple log segments (or virtual logs) is handled automatically by SQL Server and is the foundation of the circular nature of the physical log file (see below). Each VLF is in turn composed of numerous log blocks.

VLFs are created dynamically as needed by SQL Server. While we as administrators are not able to set a static size for these file divisions, we can manage the auto-growth settings of the log file, which will in turn impact the size and number of the VLFs that SQL Server creates.

4.     Log Sequence Number (LSN)

Within the database’s transaction log, the chronological consistency of transactions is preserved across the VLFs through LSNs, or log sequence numbers. These numbers allow SQL Server to arrange the transactions in their correct chronological order for the purposes of rollback and recovery despite the potential for having been written to different VLFs.

The LSN is a concatenation of the VLF ID, the Log Block ID, and the Log Record ID separated by colons. You can see examples of the LSN by querying the sys.dm_db_log_info view and looking at the [vlf_create_lsn] field:

5.     Minimum Recovery LSN (MinLSN)

The MinLSN, or minimum recovery LSN,is the LSN that would be needed for a full recovery of the database. Put another way, the MinLSN is the first LSN in the sequence of the active section or tail of the log, which is the section of the log that has not yet been written from memory to disk by a checkpoint operation.

How does SQL Server logging work?

6.     Checkpoint Process

Checkpoints are the SQL Server operations by which database modifications that are being stored in memory are written to disk. Checkpoints are usually triggered by a log backup or automatically by the recovery interval setting of the database.

The active section of the log that has not yet reached a checkpoint can never be truncated, so paying attention to the recovery settings of each database helps to manage checkpoints and transaction log truncation optimally.

7.     Cyclical Logging

The SQL Server transaction log can be thought of as a revolving log file.

Consider that the physical log is allocated a designated amount of space in the database properties when it is configured. As mentioned above, the transaction log is made up of smaller segments (VLFs) that SQL Server writes to as part of its internal logging process – adding a new VLF each time the last one fills up. This process is internally controlled by the application.

If the physical transaction log is never truncated, it will continue to grow, adding new VLFs as it grows, until it uses up all the space available to it and SQL Server fails.

If, however, the transaction log is truncated, older VLFs containing transactions with LSNs prior to the MinLSN will be cleared. When those files are truncated, the space that was being used by the deleted log records will be released and made available for new log records to be created as more transactions are committed. This circular process allows the overall size of the log to remain consistent despite ongoing transaction logging.

This log-and-truncate cycle will continue successfully unless the logging rate exceeds the truncation rate. Note once again, however, that truncation of the transaction log cannot occur in full or bulk-logged recovery models if log backups are not taken first.

For this reason, administrators should pay careful attention to the recovery and log file settings to avoid unnecessary disruptions in the SQL Server environment.

What else should you know about SQL Server transaction logs?

In an upcoming post, we will discuss how to troubleshoot some common issues with SQL Server transaction logs and give some best practices for transaction log management.

When configured carefully, the logging process within SQL Server can operate seamlessly for years without DBA intervention.

Here are some resources that contain more information about some of the topics we touched on:

• Log Blocks
• LSNs
• Virtual Log Files and Growth Algorithms
• SQL Server Checkpoint Process
• Transaction Log Backups and Log Shrinking

Want to work with The SERO Group?

If you’re concerned about your backup strategy, or more to the point, your ability to restore a critical database, contact us. We’re happy to help.

The post Anatomy of a SQL Server Transaction Log appeared first on The SERO Group.

In this post, we will address both of these important questions.

What is a SQL Server Recovery Model?

In SQL Server environments, the recovery model setting is a database-level configuration that defines how transaction logs and backups are managed for each database. This setting determines how the transaction log is managed and whether point-in-time recovery options are available for restoring that database.

There are three recovery model settings in SQL Server: Simple, Full, and Bulk-Logged. Let’s look at each.

Simple Recovery Model

With the Simple Recovery Model, SQL Server essentially relieves you from having to manage the transaction log. It takes care of it for you. SQL Server automatically truncates the transaction at regular intervals. Scheduled backups of the transaction log are not taken.

As such when a database is set to the Simple Recovery Model, restoring transactions since the last full or differential backup is not an option. This means that data between backups has the potential to be lost in a crisis.

The Simple Recovery Model is designed for situations where point-in-time recovery is not of crucial importance.

When should you use the Simple Recovery Model?

Simple recovery should be used when the risk of data loss between scheduled full or differential backups is acceptable, and straightforward backup and recovery processes are preferred. Development, test, and non-critical production databases are often set to simple recovery.

            Pros

• Simplified backup process
• Straightforward recovery procedures
• Typically requires less storage for transaction logs

            Cons

• Recovery is limited to the time of the last full or differential backup
• Potential data loss of any transaction since the last data backup
• Cannot be used with log shipping, Always On Availability Groups, or Database Mirroring

Full Recovery Model

With the Full Recovery Model, transaction logs are retained until they are explicitly backed up. They are not automatically managed by SQL Server. The logs are only truncated after a log backup has been taken. Note: this means that regular transaction log backups are not just an option but a requirement in this model to prevent the log from growing too large. If the transaction log is not explicitly backed up, it may continue to grow unbounded and fill all available disk space. (You can set a maximum size for the transaction log to prevent it from filling the disk, however once that maximum size is reached the database will not be available for transactional purposes.)

The Full Recovery Model enables point-in-time recovery of your critical databases.

When should you use the Full Recovery Model?

The Full Recovery Model is most appropriate for critical databases where potential data loss between backups must be minimized.

It is also appropriate to use this model when using log shipping, Always On Availability Groups, or Database Mirroring. Likewise, it can be used for databases with frequent transactions since the frequency of log backups can be scheduled manually as needed. Practice Management Systems in healthcare, Enterprise Resource Planning Systems in manufacturing, and financial systems in FinTech applications typically use the Full Recovery Model.

            Pros

• Fully logged to support point-in-time recovery
• Offers the highest level of protection against data loss for mission-critical data

            Cons

• Requires more complex backup and recovery processes, including transaction log backups
• The transaction log can grow rapidly if not managed properly

Bulk-Logged Recovery Model

The Bulk-Logged Recovery Model is a hybrid approach to database recovery. It is designed for databases that see frequent bulk operations, like bulk inserts or index rebuilds that would produce large transaction logs, and whose RPO (recovery point objective) is shorter than the time interval between full or differential backups.

Like the Full Recovery Model, the Bulk-Logged Recovery Model requires explicit log backups. However, unlike the Full Recovery Model, Bulk-Logged recovery uses minimal logging while bulk operations are being performed, reducing the rate at which the transaction log grows and improving performance for these operations.

Full transaction logging of the bulk-operation is still retained in the log backup and can be leveraged for point-in-time recovery. However, transactions occurring during minimally logged bulk operations are vulnerable to loss until the operation is complete and the next log backup has been taken.

When should you use the Bulk-Logged Recovery Model?

The Bulk-Logged Recovery Model is most appropriate for databases (like in data warehouses or in data marts) where large-scale data operations are frequent.

            Pros

• Fully logged to support point-in-time recovery like the Full Recovery Model
• Offers performance improvement and reduced risk of log growth during bulk operations

            Cons

• Requires more complex backup and recovery processes, including transaction log backups
• There is limited point-in-time recoverability during bulk operations, and restoring to the most recent log backup would be necessary in the event of a failure

More information

Choosing the appropriate SQL Server Recovery Model for each of your database requires an understanding of data it contains, as well as the business’ RPO for the data. For example, if the business can afford to lose up to 24 hours of data, the nightly backups and a Simple Recovery Model may be a good approach. On the other hand, if minimal data loss is acceptable for a database, the Full Recovery Model with combination full backups, differential backups, and frequent log backups will likely be the model of choice.

Do you have questions about SQL Server Recovery Models or your recovery strategy? Are you unsure of what recovery models your SQL databases are using?

Here are some additional posts that may help:

• How to Align Your SQL Server to Your RPO and RTO Goals
• Where to Start with Disaster Recovery in SQL Server
• Using VM Snapshots to Backup SQL Server?
• Why performing database restores before a crisis strikes is a good idea

Or reach out to us! We would be happy to hop on a call or to set up an initial health check to help you perform an assessment. Schedule a call with us to get started.

The post What is a SQL Server Recovery Model? appeared first on The SERO Group.

In the last post, we talked about applying the Kaizen approach to data management to achieve a culture of continuous improvement on our data teams.

In this post, we will use the KonMari method of simplification, recently made famous by Marie Kondo, as a lens for considering what to keep and what to purge in our business data repositories.

As always, we will also give some best practices for how to establish policies around archiving and purging your company data.

Data Lifecycle Management: Archiving and Deletion

As mentioned in previous posts, the final phases of Data Lifecycle Management are Archiving and Deletion.

These phases help to ensure that we keep and maintain only that data which is required for our business. But how do we determine what to keep active, what to archive, and what to purge?

We can apply some concepts from the KonMari simplification method to our data strategy here to help us decide.

Rule 1: Make the Commitment

Kondo states that the first step in KonMari is committing to achieving your goal. This may seem like an obvious first step for any endeavor, but many companies fail to establish a data retention strategy.

It is not uncommon for businesses to take the approach that keeping all data (sometimes even in a “hot”, or readily accessible, repository) is the way to go. This tactic usually stems from either:

1. an explicit belief that you cannot go wrong with keeping too much historical data
   or
2. from having no capacity to prioritize a retention strategy.

Either way, the “keep everything” strategy is misguided for 3 reasons.

First, the more data you keep, the more time it will take to recover in the event of a crisis.

Crisis can take the form of:

• a lawsuit or an audit that requires retrieval of specific information

or

• a natural disaster, human error, criminal activity or another event that demands restoration of data to a particular point in time.

In any case, restoration time is critical during these events. The more time it takes to retrieve the required data from your archive, the longer it will take for the business to recover.

Second, while data retention is a necessity, it is also a liability and entails responsibility.

Businesses must take the responsibility to respect consumer privacy rights very seriously. Part of this responsibility entails keeping consumer data for no longer than is required or for any purpose other than that for which the consumer gave consent. Even if your company does not fall under the regulatory jurisdiction of privacy laws like GDPR, CCPA, or PIPEDA, the business is nevertheless liable for securely and responsibly maintaining its consumer data.

With any data that is retained comes the possibility that it could be stolen, leaked, or misused. This risk is unavoidable, but preserving unnecessary archives of historical data is a liability that ought to be avoided.

Third, “Data stores don’t grow on trees…”

A well-crafted data strategy can reduce the financial cost of maintaining your data repositories, but increasingly large data stores cost the business money nevertheless.

There are also performance, time, system resource, and opportunity costs associated with maintaining large data stores.

So, in short – make the commitment to tidy up your unwieldy data repositories!

Rule 2: Imagine Your Ideal

Compliance regulations do some of the work of envisioning the ideal for us in the data world. Still, take the time to consider the ideal composition of your data repositories. Doing this can help you to think strategically about what should be kept, how and where to keep it, and for how long.

As mentioned in the previous post in this series, consider both regulatory requirements and the needs of the business for reporting, analytics, and strategic planning when determining what to keep. Consult business leaders and business analysts about what data is needed and for how long. You can even create a formal data contract for critical data elements in your business.

Rule 3: Finish Discarding First

Obviously, deleting data should always be done with extreme caution and forethought.

Nevertheless, once you have performed an audit of your data repositories and have determined your retention strategy, you should begin implementation by purging unnecessary data. We will discuss more about how to perform this action safely and according to best practices below.  

Rules 4 and 5: Progress by Category and in the Right Order

For the purposes of purging and archiving data, we should be thinking in a criticality/age matrix like the one below – beginning in the upper left corner and working down and to the right.

You should make incremental passes across departments in these stages, beginning with the oldest and least important data in each departmental area.

Rule 6: “Does it Spark Joy?”

Ok, ok – I admit this iconic final question from Kondo is much less suitable to data retention. However, there still may be an important (if less-than-perfectly-measurable) question that should be asked by data teams before coding a delete.

Does a business leader strongly prefer to retain certain data despite the lack of any clear regulatory or business-driven reason for doing so?

If so, keep it…unless doing so presents a serious risk or concern. If you feel there is a serious risk, continue to voice your concerns. Otherwise, just wait and continue to get clarification.

Now for some best practices…

Keep in mind that the practices listed here do not include the critical practice of backing up your production data, which has been discussed in previous posts (here, here, and here). Always make sure that backups are in place before beginning to archive or delete data.

1.     Replicate and archive data “in-flight”.

Archiving and/or replicating your data at various points in your pipeline is a best practice. Process failures in data pipelines are not uncommon, and you need to be able to recover data from earlier stages of the pipeline if you need to reprocess the data.

3 common examples of this practice, which should include purging data after an established retention period, are:

• Moving imported files to an archive folder
• Replicating transactional databases to a staging database before further processing by downstream systems
• Staging imported API data in its own table or database before integrating with internal systems.

2.     Archive in cold storage and protect the archive.

Consider how you will store data that has served its immediate purpose and has been determined to be a candidate for long-term storage. There are pros and cons to each method, and a combination of archiving methods may be appropriate for your different data sets. Here are some options and considerations.

• Onsite physical storage
  • Pros: ease of access in an emergency, familiar technology
  • Cons: vulnerable to tampering, theft, and physical damage/natural disasters
• Offsite storage (tape, optical disk, magnetic hard drives)
  • Pros: well-established, relatively inexpensive, usually more secure than onsite
  • Cons: slower recovery times
• Cold cloud storage
  • Pros: speed and ease of recovery, alleviated burden of maintenance, built-in security, inexpensive
  • Cons: potentially less familiar to established IT departments than traditional methods
• Data lake
  • Pros: accessibility, speed and ease of recovery, suitable for use with emerging technologies, built-in security
  • Cons: less-established, steeper learning curve, potentially expensive, requires careful governance of user access

3.     Don’t forget critical data that is managed by third parties.

Over time, businesses may find that a significant body of company data resides in data stores managed by third parties. Since maintenance has often been delegated to vendors in these cases, archiving this data is sometimes overlooked.

While there may be an ability to set a retention policy, it can be very beneficial for many reasons to work with your vendors to set up an extraction process to archive a copy of your data from these systems into your own repositories as well.

4.     Establish retention schedules and procedures for purging data.

Business, regulatory, and legislative needs dictate what should be saved and for how long, and these may differ between data sets. Establishment of policies and procedures for deleting data will ultimately be the responsibility of data owners.

These policies should address the following areas:

• Who is authorized to purge data?
• In what maintenance windows can this process occur since deletion can go slowly and data processing jobs/replication must be disabled?
• How will notification be given to the business?
• What validation and integrity checks must be in place?
• What rollback procedures will be used if necessary?

Just the tip of the iceberg…

There is much, much more to say about all these topics.

If you have made it as far as committing to cleaning up your data but the rest seems overwhelming, never fear! There are many vendors that are happy to help with all levels of assistance.

If you have a good handle on your archiving and deletion processes but would like assistance with a SQL Server implementation of them, reach out! We are here to help.

The post Archiving and Deletion Strategy…KonMari for Data Management? appeared first on The SERO Group.

This refrain is more common than you think in IT departments of all sizes. Or maybe you live that reality every day and are fully aware that clunky, error-laden processes eat away at your team’s efficiency (and morale).

Data management and continuous improvement may sound like they should always go together, but they often don’t.

Many times our data management practices involve too many business-critical data processes that break regularly and need to be improved, but we have no time to make the needed improvements because there are so many data processes that need to be “managed” (a.k.a. remediated regularly).

So, how can we break this cycle?

Welcome back to the second of three posts on how to refine your strategy for Data Lifecycle Management (DLM)!

In this post, we will focus on Data Management as the second of the three DLM stages: Data Collection, Data Management, and Data Deletion.

Kaizen for Data Management

The Kaizen approach, famously championed by the Toyota corporation, suggests that small organizational changes can lead to a culture of continuous improvement. This culture will ultimately lead to better processes, greater efficiency, improved outcomes, and increased morale.

The Kaizen Institute states,

“As part of the corporate culture, continuous improvement becomes an ongoing process integrated into the organization’s daily activities. Employees are encouraged to challenge the status quo, suggest ideas, and implement improvements. Continuous learning and development are valued, and mistakes are seen as growth opportunities.”

This means that adopting a Kaizen approach to your data management strategy can be a lever for driving a continuous improvement culture on your data team without sweeping, drastic changes.

Small improvements to existing processes can slowly bring significant reduction in process failures, and improvements in efficiency, accuracy, and team morale. (Here is a short article about applying a Kaizen approach in an IT context.)

So, how and where can we improve our data management?

Where should you look to start identifying small improvements that might be implemented?

Consider the areas below with your team. Most likely you will find that you are very strong in some areas, but perhaps there are areas that have not been addressed at all. Start with the lowest hanging fruit, and bit by bit you will find that you are slowly filling the gaps and addressing the technical debt that every established data team faces.  

First, analyze your data structures.

The applications, tools, and data processes in place for your company will impact the data structure that needs to be in place for it to be usable. Unfortunately, these requirements rarely align.

When you think about the flow of your data, think about consistency of format and type. As data flows into your system, it is often riddled with discrepancies in format, data type, and even the information it contains (but we will save that for another post).

As your data flows downstream toward the consumer, it should become more and more aligned in these areas. Why? Because the more points of contact that your technical teams must have with it (to transform it for particular use cases, etc.), the more points of failure you can have.

Strategic policies and governance and centralized data management can really help, but you don’t need an operational overhaul to improve!

In line with a Kaizen approach, try encouraging small changes in these areas:

Establish data standards

This will be an ongoing process. You will want to give thought to what your core standards should be, especially for mission critical data elements like identifiers, account numbers, etc., since these are more difficult to change once processes are mature. However, your standards will expand and refine as your business matures its data processes.

Adopt an enterprise modeling tool

Document and catalog your data standards using a modeling tool. Include all the metadata associated with your data objects and their relationships. The business will use the resulting documentation at every level (system administration, development, business analysis, and report consumption) for understanding and interpreting the data.

Transform your data with consistency

Wherever your transformation layer lives in your processes (and hopefully there are as few of these as possible), always architect toward your established data standards.

Establishing governance and centralized management can really help here, but feel free to start small! Apply these principles to new processes and only to established processes as they require other changes. Encourage a culture that celebrates these improvements and looks for opportunities to make things better.

Implement database source control

That’s right – employ a source control process for your database objects. Many companies do not take this step. However, having source control in place does not only protect your team from losing important data objects. It can also help ensure that new structures follow established standards when code reviews, pull request approvals, and other best practices are in place.

Structure your deployment process

Lastly, establish protocols around deployment. Some options include:

• Creating a deployment cadence that uses established deployment windows
• Setting up a change advisory board for reviewing changes before approving them to be deployed to production
• Designating deployment managers that are responsible for deploying code
• And, of course, you can always automate your deployments! Just be careful to include the appropriate guardrails.

Remember – slow and steady wins the race with continuous improvement.

Second, evaluate your data pipelines.

Outside of data structure, there are other data process considerations that need to be evaluated as well.

Accuracy & Reliability

• Are your data ingestion and replication processes accurate and reliable?

Sometimes when evaluating our pipelines, we find that issues with error handling, purge processes, SFTP, APIs, replication, logging or any number of other processes are causing duplicative, inaccurate, incomplete, or undelivered data transfers. Look out for these and correct them as you find them.

Maintenance & Scalability

Also, ask yourself these questions:

• Are you frequently stretching the limits of any of your allocated hardware, VMs, databases, or network resources?
• Are any of your system resources in need of upgrades or patching? Are you missing protocols to ensure that these are completed?
• Are there other systems, applications, technologies, or vendors that might suit your current or projected needs better?
• Are your data processes too slow? Do they struggle with the amount of data that must be processed by them?

If your answer to any of these questions is “yes”, then you have opportunities for improvement (slow and steady…).

Third, never forget about security with data management.

Security should always be top of mind when considering your company data. Here are some areas to evaluate.

Security – External

Review the security around the infrastructure supporting your company’s data processes for points of external connection. Pay particular attention to any processes that utilize third party tools or that export or extract data to/from external sources.

Security – Internal   

For internal sharing and usage, security measures should be concerned with careful provisioning of access to data and systems. For lower-level systems, be sure to mask or de-identify any sensitive data.

Further, for sensitive or confidential data, give careful consideration to protecting against any intentional or unintentional data leaks. Areas to consider creating policies around include:

• Unsecured physical devices or paperwork
• Keeping only what data is necessary
• Emailing sensitive data
• Downloading data to personal devices
• What to do if a suspected data breach has occurred

Is your head spinning? Don’t worry!

Remember that data management is an ongoing process of continuous improvement, and we will delve into many of these topics more deeply in upcoming posts.

In the meantime, if you have a pressing need and could use some help detailing a roadmap, let us know! We love to help empower continuous improvement with our clients.

“We cannot become what we want to be by remaining what we are.”

– Max DePree

The post Data Management Strategy: A Kaizen Approach appeared first on The SERO Group.

Welcome to the first of three posts on how to refine your strategy for data lifecycle management. In this post, we will look at how to evaluate your data collection processes for improvements.

Data Collection in Data LifeCycle Management (DLM)

As has been noted in a previous post on the difference between Data Lifecycle Management (DLM) and Information Lifecycle Management(ILM), there are fundamentally 3 phases of Data Lifecycle Management (DLM) into which all physical data-related tasks fall: Data Collection & Creation, Data Management, and Data Deletion.

As an IT leader, there are two important exercises you should perform to evaluate your data collection strategy. By performing these, you will produce living documents that should guide how your company creates, ingests, and consumes data now and in the future.

First, perform a data collection audit.

The first step in evaluating your DLM processes is to gain a complete understanding of the data collection processes your company is currently using. The best way to do this is through an internal audit.

Your data collection audit should include answers to the following questions:

• Where is data coming into your systems (websites, transactional systems, vendors)?
• What systems or processes are used to create or collect data (software, web forms, APIs, FTP)?
• What data formats are being leveraged by these processes (SQL, JSON, CSV, XML)?
• What security, threat mitigation, backups, and archiving processes are in place for these processes and data stores?  (Here is a good summary of what to look for.)

Finally, examine the information you gathered through a strategic lens. Look for vulnerabilities, inefficiencies, and pain points in your processes. Then work with your team to devise a strategic plan and implementation timeline for achieving improvements in these areas.

Second, create a data tracking plan.

Now that you have audited your data collection processes, you should give thought to why you are collecting the data that you are and what data needs to be collected. Consult business analysts in your company about which metrics they would like to track. This will help you understand what data points need to be collected. Likewise, find out what government regulations dictate about what data should be collected and retained.

Ask questions like:

• Is the right data being collected?
• Are there any missing data points?
• Are you collecting irrelevant or duplicated data?

Bridge the Gap

Undoubtedly, it is tricky to bridge the gap between the business, which has ideas about what data they would like to track, and the technical team, who knows how to track it. A data tracking plan is a tool that can help with this.

While some people strictly define what a data tracking plan must consist of, a simple plan is often sufficient. Your data tracking plan defines your primary business objects (customers, products, stores, etc.) and the metrics or events surrounding them that your business would like to have more information about.

Before spending time creating your own, take a look at the many templates available to get you started. Here is a link to an evaluation of a few free templates to start your research.

Create the plan

Once you have your template, start your internal planning discussions with questions like:

• What core business objects are we concerned with?
• What metrics do we care about for those objects (that is, what do we want to track about them)?
• Why do we want to track these metrics?
• What data needs to be collected to obtain these metrics and how will it be defined?
• Where can the data be obtained? Do we already collect it?
• Who will govern the information once we have it?
• Who will manage the data collection?
• What format does the data need to be in to be useful?

It can certainly be challenging on many fronts for the business and IT to come together to create a data tracking plan. However, facilitating this will be well worth the effort in terms of the clear data strategy objectives that will be produced. Avoiding the costs associated with misguided data projects will more than outweigh the time and energy spent in coordinated planning.

Finally, update your data strategy, and implement changes.

Once you have assessed your data collection processes and have identified improvements, you’ll need to assign priorities to your findings. Work with both the business and your technical team to set these priorities, as well as to build a roadmap for implementation.

As can be seen, the information you have gathered through cross-functional cooperation and through using these tools will help you to make a strong case to business leaders for the importance of these strategic improvements.

Want to learn more?

Looking for more information about Data Strategy and how it can help align IT and business goals? Check out these posts.

If you’d like to learn more about how we approach Data Strategy, or if you have some concerns about your SQL estate, give us a call. We can help.

The post Data Collection: Two Key Tools to Improve Your Data Strategy appeared first on The SERO Group.

Recently, we began a series of posts on data strategy by asking whether your company data is an asset or a utility. We then uncovered 10 Data Storage Considerations to Improve Your Company Data Strategy. In this post, I will argue that recognizing the distinction between DLM and ILM is important for IT leaders to help their companies get the most out of their data.

Now, the difference between Data Lifecycle Management (DLM) and Information Lifecycle Management (ILM) might initially seem like hair-splitting. However, while DLM incorporates management policies for the physical aspects of data as data (type, size, location, age, etc.), ILM addresses the content of the data as information (its accuracy, reliability, sensitivity/confidentiality, etc.).

Here are some of the differences and why both are important to consider.

Data Lifecycle Management

At its highest level, DLM addresses data creation, data management, and data deletion.

Within these categories, there are many other elements of data management that are included as well:

• Creation / Collection
• Classification
• Redundancy
• Duplication
• Integrity
• Usage and Availability
• Sharing
• Storage
• Security
• Archiving

All these data process elements when viewed through DLM are fundamentally concerned with the physical aspects of managing data. Failing to address these areas in your data strategy is to gamble with the health of your data structures and the value of your company data.

Information Lifecycle Management

On the other hand, ILM addresses issues associated with the information the data contains. ILM establishes policies that manage the data quality, business relevance, regulatory compliance, and legal liability of the data.

Many of the specific areas addressed by ILM establish protocols for processing data in a way that ensures data accuracy, reliable delivery, protection of sensitive information, and compliance with data privacy laws. Some elements of ILM can include:

• Data De-Identification / Masking
• Data Quality Frameworks & Audits
• Development and QA Environment Refreshes
• Source Control
• Master Data Management
• Classification and Governance
• Sharing & Usage
• Security
• Regulatory Compliance Audits

While many elements of the data lifecycle have relevance in both management models, they are viewed through a different lens in each model.

So…Who Cares?

So, why does recognizing this distinction matter? Will your business really suffer without it?

The answer is that it matters because if this distinction is not recognized, the two sets of management policies can get conflated, with one getting largely ignored. When either DLM or ILM is neglected, important elements of the data lifecycle can appear to be fully managed when in fact they are only partially addressed.

Perhaps even more importantly, when IT leaders do not see these management policies as distinct, they risk missing out on the benefits that come with using both together, like:

• Improved system performance
• Increased availability and accessibility of data
• Improved data quality
• Increased data consistency across the organization
• Improved recoverability
• Increased security
• Increased user satisfaction
• Controlled costs
• Improved regulatory compliance

In upcoming posts, we will unpack these models further while exploring more ways to improve your data strategy.

Want to work with The SERO Group?

Would you like some outside input in any of these areas? We love to work alongside IT leaders and their teams to help them establish the use of best practices in their data environments.

If that’s something you’d like to learn more about, let’s have a conversation.

The post Are Information and Data Lifecycle Management Processes Different…and Who Cares? appeared first on The SERO Group.