What if you could retire two aging SQL Server 2016 failover clustered instances, migrate 1,200+ databases to SQL Server 2022, and slash storage spend by 84%—all in the length of a weekend maintenance window?

In this post, I’ll tell you exactly how we did it for a client using AWS.

We had multiple goals for this migration:

• Move to SQL Server 2022 and Windows Server 2022
• Reduce Amazon AWS storage costs
• Minimize the downtime required for the migration

The environment consisted of two SQL Server 2016 Enterprise failover clustered instances hosting over 1,200 production databases. As you may know, mainstream support for SQL Server 2016 ended on July 13, 2021, with extended support ending on July 14, 2026 (which is approaching quickly). This post explains how we successfully migrated this complex environment with less than 3 hours of planned downtime while simultaneously reducing infrastructure costs by transforming their storage architecture and backup strategy.

Key Results:

• 1,200+ databases migrated with minimal business disruption.
• Infrastructure storage costs reduced by 84% (estimated).
• SQL Server platform modernized from 2016 to 2022.

Client Profile

Industry: Education software

Environment:

• Two SQL Server 2016 Enterprise failover clustered instances in Amazon AWS
• FSx file share volumes for database storage
• FSx file share for backups
• Multi-subnet configuration
• 1,200+ production databases ranging from 1GB to 3TB in size
• 24/7 business operations with minimal tolerance for downtime

The Challenge

Business Drivers:

The client approached us with several business requirements:

Cost Reduction: Their existing FSx storage and backup architecture was consuming an unsustainable portion of their IT budget.

Technology Modernization: SQL Server 2016 was approaching the end of extended support, requiring an upgrade path to maintain vendor support and security compliance.

Minimal Disruption: With 1,200+ production databases supporting critical business functions, any migration solution needed to minimize downtime.

Technical Challenges:

The technical landscape presented several significant obstacles:

FSx Storage Cost and Performance

The client’s reliance on Amazon FSx for Windows File Server provided the necessary shared storage functionality for their SQL Server failover clusters, but at a premium price point. The FSx architecture also introduced additional network latency for I/O operations, affecting database performance.

Complex Multi-Subnet Configuration

The existing environment leveraged a multi-subnet configuration, adding complexity to any potential migration solution.

Database Scale and Diversity

With over 1,200 databases that vary dramatically in size, usage patterns, and business criticality, a one-size-fits-all migration approach would be insufficient.

Backup Architecture Limitations

The existing backup strategy relied on additional FSx storage, increasing the storage costs while creating management overhead.

Strict Downtime Constraints

The business could tolerate a maximum of 8 hours of downtime, with a strong preference for staying under 4 hours to minimize impact on operations.

The Solution

After a comprehensive analysis and planning, we designed a phased approach that would transform the client’s database infrastructure while meeting their strict requirements for minimal disruption.

Solution Architecture

We architected a modern SQL Server environment on AWS with the following components:

1. Platform Upgrade:
   • SQL Server 2022 Enterprise Edition
   • Leveraged latest features and extended support lifecycle
2. Storage Transformation:
   • From FSx file shares to Multi-Attached EBS volumes
   • io2 Block Express volumes for critical data files
   • Optimized configuration for SQL Server workloads
3. Backup Modernization:
   • Implement S3-based backup strategy
     • Direct SQL Server native backup to S3
   • Intelligent lifecycle policies for cost-effective long-term retention
   • Elimination of dedicated backup storage infrastructure
4. High Availability Design:
   • SQL Server 2022 FCI
   • Modern monitoring integration

Implementation Approach

We divided the implementation into distinct phases to manage risk and ensure successful outcomes.

Phase 1: Development Environment Migration

We began with the client’s development environment to validate our approach and identify potential issues. We worked with the client’s internal system admins to complete the EC2 instance and Windows Server Failover Cluster builds. This allowed us to focus mainly on the SQL Server FCI components:

1. Infrastructure Provisioning:
   • Deployed new EC2 instances optimized for SQL Server 2022
   • Configured multi-attached EBS volumes
   • Established networking and security configurations
2. Initial Testing:
   • Validated storage performance
   • Confirmed failover functionality and timing
   • Verified backup and restore operations to S3
3. Migration Dry Run:
   • Executed migration process on a subset of development databases
   • Measured migration times
   • Refined automation scripts based on observations

Phase 2: Production Environment Preparation

With lessons from the development migration incorporated, we prepared for the production migration:

1. Pre-Migration Activities:
   • Comprehensive database inventory and dependency mapping (i.e., agent jobs, applications, etc.)
     • Planning, communication, and task tracking were crucial to the overall success of the migration.
   • Database health checks to identify potential migration blockers
     • Including utilizing Microsoft’s Data Migration Assistant to ensure database compatibility
   • Implementation of enhanced monitoring
2. Automation Development:
   • Custom PowerShell script creation for orchestrating the migration
   • Integration with DBATOOLS for database backup and restore operations
3. Infrastructure Deployment:
   • Production SQL Server 2022 cluster provisioning
   • Multi-attached storage configuration and testing
   • S3 bucket preparation with appropriate security and lifecycle policies
   • Tested access to the current backup location from the new SQL Server 2022 instances (permissions and network access)

Phase 3: Production Migration Execution

The production migration followed our carefully orchestrated plan:

1. Pre-Stage Initial Data:
   • Performed initial data copy without taking systems offline
   • Utilized a combination of native Full, Differential, and Log backups
   • Leveraged the rebuilt PowerShell script to perform the initial full restore and differential restore
   • Restored log backups every 30 minutes
   • Scheduled restores via the SQL Server Agent on the new SQL Server 2022 FCIs
   • Established baseline synchronization to minimize final transfer time.
2. Migration Window Execution:
   • Implemented application maintenance mode at the start of the maintenance window, blocking access
   • Executed the final data synchronization using our custom migration script
     • Tail-log backup was used to capture all transactions and place the database on the SQL Server 2016 instances into “recovering” status
       • This accomplished two important goals:
         • Capture all transactions that have occurred since the last transaction log backup
         • Place the database into a “recovering” state so that new connections were not allowed in the old environment
   • Performed database restores on the new SQL Server 2022 instances using all transaction log backups, including the final tail-log backup
     • Databases were then recovered and placed into a multi-user read-write state
   • Raised the database compatibility level to 160
   • Changed the database owner
   • Validated each database post-migration
   • Initiated post-migration full backups
3. Cutover:
   • Updated connection strings and DNS entries
   • Performed final validation testing
   • Opened systems for production use

The Technology

Multi-Attached Storage Architecture

The shift from FSX to multi-attached EBS volumes represented one of the most significant technical innovations in this project. We configured the volumes with performance characteristics matched to their specific workloads:

• Data volumes: 3,000 IOPS
• Log volumes: 3,000 IOPs
• TempDB volumes: Amazon EC2 instance store volume (local to each EC2 instance)

This approach delivered both performance improvements and significant cost savings compared to the FSx solution.

Custom Migration Automation

The scale of this migration—over 1,200 databases—demanded sophisticated automation. We developed a custom PowerShell script that utilized multiple commandlets from the DBATools module.

DBATools Integration

While our custom script orchestrated the overall migration process, we leveraged the power of DBATools—an open-source PowerShell module for SQL Server management—for many of the core database operations. This combination of custom automation with proven community tools provided the perfect balance of flexibility and reliability.

Key DBATools functions employed included:

• Get-DBABackupHistory to retrieve the latest backup information for each database.
• Restore-DbaDatabase for restoring the initial full, differential, and all t-log backups.

The Results

The migration was completed successfully with impressive metrics:

Cost Reduction

• 84% reduction in monthly storage costs

Business Impact

• Minimal disruption: Total downtime of only 2 hours and 55 minutes, well under the 8-hour target
• Extended support lifecycle: Movement to SQL Server 2022 provides support through 2032
• Performance headroom for growth: New infrastructure accommodates projected 55% database growth

Migration Metrics

• Total databases migrated: 1,235

Key Lessons Learned

Throughout this project, we gained valuable insights that would benefit any organization undertaking a similar database migration:

1. Thorough database inventory is critical
   • Understanding dependencies between databases was crucial
   • Early identification of current database sizes allowed for proper allocation of storage to support the migration.
2. Storage strategy has an outsized impact
   • Moving to multi-attached EBS volumes delivered both cost and performance benefits
   • SQL Server 2022 introduced the ability to backup directly to S3, delivering significant storage savings vs. FSx
3. Automation is non-negotiable at scale
   • Manual migration for 1,200+ databases not possible within time constraints
   • Investment in custom automation paid dividends in reliability and speed
4. Pre-staging reduces risk
   • Copying the bulk of data before the migration window dramatically reduced the final migration time
   • Delta synchronization proved highly efficient using t-log backups
5. Multi-phase approach validates methodology
   • Starting with the development environment allowed refinement of techniques
   • Lessons learned in initial phases improved production migration execution

Conclusion

This post demonstrates how even the most challenging database migrations can be accomplished with minimal downtime when approached with careful planning, appropriate technology choices, and custom automation. By transforming the storage architecture and leveraging modern AWS services, we not only successfully migrated 1,200+ SQL Server databases but also delivered substantial cost savings and performance improvements.

For organizations facing similar challenges with aging SQL Server environments, this project provides a proven blueprint for success—combining the best of cloud infrastructure, modern database platforms, and sophisticated automation techniques.

The client now enjoys a modern, cost-effective SQL Server environment that provides room for growth while reducing ongoing operational costs. Most importantly, they achieved this transformation with minimal disruption to their business operations—proving that with the right approach, large-scale database migrations don’t have to be painful or disruptive.

Want to work with The SERO Group?

Want to learn more about how The SERO Group helps organizations take the guesswork out of managing their SQL Servers? Schedule a no-obligation discovery call with us to get started.

The post How We Migrated 1,200+ SQL Server Databases in Under 3 Hours (While Cutting Storage Costs by 84%) appeared first on The SERO Group.

1. Recovering after a database outage
2. Database migrations

What is a tail-log backup?

Tail-log backups capture transaction log records that haven’t been backed up yet and set the database into a “restoring” state. Think of it as the last transaction log backup taken right before a database is restored, moved, or damaged. Its primary purpose is to capture the “tail” end of the log (any transactions that occurred since the last regular log backup to prevent data loss).

When are tail-log backups important?

Scenario 1: Preventing data loss after a failure

Imagine that you schedule transaction log backups to occur every 15 minutes. At 10:05 AM, the server hosting your database experiences a critical failure, taking the database offline. Your last log backup was at 10:00 AM. This means that transactions that have occurred between 10:00 AM and 10:05 AM would be lost without a tail-log backup.

The tail-log backup bridges this gap, capturing those final crucial log records and allowing you to restore the database to the exact point of failure. Depending on the damage, you may need to add additional options for this backup to succeed. See here for those options.

Scenario 2: Supporting a smooth database migration

If you’re migrating a database to a new instance, tail-log backups can be beneficial as well. Let’s say you have a migration coming up from SQL Server 2016 to a new SQL Server 2022 instance. You’ve pre-staged the database on the new instance by restoring the latest Full and have been applying transaction log backups periodically without recovery throughout the week leading up to migration day. On migration day, you need to ensure the following happens:

1. All transactions are captured up to and right after the maintenance window begins.
2. Ensure the database on the old instance can no longer accept new connections. You don’t want new transactions landing here by accident.
3. Restore the final transaction log backup on the new instance and recover the database.

A tail-log backup can help in this scenario as well. You see, once you perform a tail-log backup, the database you performed it on is placed into the “restoring” state, which doesn’t allow connections. This ensures no future transactions can be written. Just in case a connection string wasn’t updated somewhere during the migration (I’d personally rather the connection fail vs. being able to connect to the database that is no longer in use). Once the migration is complete, including testing and validation, the old database can be dropped.

Let’s take a look at the requirements needed before you can utilize tail-log backups.

Requirements for tail-log backups

To perform a tail-log backup, the following conditions must be met:

Recovery model

The database must be in the FULL or BULK_LOGGED recovery model. Tail-log backups are not possible (or needed) in the SIMPLE recovery model, as the transaction log is automatically truncated.

Prior full backup

At least one full database backup must have been taken previously.

Log file accessibility

The transaction log file (.ldf) must be accessible and largely intact, even if the data files are damaged or the database is offline.

You can determine if your database meets the first two requirements by using a query similar to the one below.

USE MSDB
SELECT distinct
	backupSet.[Database_Name],  
	s.recovery_model_desc AS RecoveryModel,
	backupmediafamily.logical_device_name AS LogicalDeviceName, 
	backupmediafamily.physical_device_name AS PhysicalDeviceName, 
	backupset.expiration_date AS ExpirationDate, 
	backupset.name AS Name, 
	backupset.[description] AS [Description], 
	backupset.user_name AS UserName, 
	backupset.backup_start_date AS StartDate, 
	backupset.backup_finish_date AS EndDate, 
	DATEDIFF(mi, backupset.backup_start_date, backupset.backup_finish_date) AS DurationInMinutes,
	CAST(CASE backupset.type 
	WHEN 'D' THEN 'Database' 
	WHEN 'L' THEN 'Log' 
	WHEN 'I' THEN 'Differential' 
	WHEN 'F' THEN 'File' 
	WHEN 'G' THEN 'Diff File' 
	WHEN 'P' THEN 'Partial' 
	WHEN 'Q' THEN 'Diff Partial' 
	END AS NVARCHAR(128)) AS BackupType, 
	--backupset.compressed_backup_size / 1048576 , 
--	backupset.backup_size / 1048576,
	ISNULL(backupset.compressed_backup_size, backupset.backup_size) / 1048576 as SIZE,
	is_snapshot,
	is_copy_only,
	GetDate() AS DateChecked --, *
FROM msdb.dbo.backupmediafamily AS backupmediafamily
INNER JOIN msdb.dbo.backupset AS backupset ON backupmediafamily.media_set_id = backupset.media_set_id
INNER JOIN master.sys.databases as s ON backupset.database_name = s.name
WHERE     (CONVERT(datetime, backupset.backup_start_date, 102) >= GETDATE() - 1)
AND backupset.server_name = @@servername  --Filters out databases that were restored from other instances.
AND backupSet.[Database_Name] = 'YourDatabaseName'
AND backupset.type = 'D'
--AND backupSet.Type = 'D'
ORDER BY StartDate --DurationInMinutes DESC

How to perform a tail-log backup (high-level)

The command is a variation of the standard BACKUP LOG statement. The key difference often lies in the options used, particularly WITH NORECOVERY or WITH NO_TRUNCATE.

Scenario 1: Database damaged/not starting (log file intact)

In this scenario, you may have lost a drive containing your data files (.mdf). You’re lucky because you’ve been following the old adage of keeping your data files and log files on separate disks so the log file is available.

If the database data files are damaged or missing, the database cannot start normally. You can attempt a tail-log backup using WITH NO_TRUNCATE. This tells SQL Server to back up the log records without trying to access the data files or truncate the inactive portion of the log, which might fail since the database is damaged.

BACKUP LOG ExampleDB
TO DISK = '\\YourBackupShare\Backup\ExampleDB_TailLog_NoTruncate.trn'
WITH NO_TRUNCATE;

Scenario 2: Planned migration

In scenario 2, you’re migrating the database to a new instance and must ensure all transactions are captured. I’ll typically switch the database into SINGLE_USER mode and kill all other connections when doing so.

ALTER DATABASE ExampleDB SET SINGLE_USER WITH ROLLBACK IMMEDIATE

Perform this step once your migration window has started and any applications, scheduled tasks, jobs, etc. have been stopped.

Perform the tail-log backup.

BACKUP LOG ExampleDB
TO DISK = '\\YourBackupShare\Backup\ExampleDB_TailLog.trn'
WITH NORECOVERY;

Here’s what happens when you run this backup.

1. The final log records are backed up to ExampleDB_TailLog.trn.
2. The ExampleDB database is put into the RESTORING state.
3. No further transactions can occur in the original ExampleDB database.
4. You can now proceed to restore any subsequent differential/log backups that you haven’t restored yet and, finally, this tail-log backup (using WITH RECOVERY) on the target server.
5. Once restored and recovered, don’t forget to place ExampleDB into MULTI_USER mode on the target server.

Tail-log backup process checklist

Here’s a quick checklist for performing a planned tail-log backup (like for a migration):

• Verify Recovery Model: Ensure the database is in FULL or BULK_LOGGED mode.
• Check Backup History: Confirm a recent full backup exists. Regular log backups should be running.
• Notify Users: Inform users of the planned downtime. Work with all other teams that depend on the database to determine a sufficient maintenance window for the migration. They’ll need to update connection strings, scheduled jobs, SSIS packages, etc., to point to the new database location.
• Restrict Access: Prevent new connections/transactions just before the backup.
• Execute BACKUP LOG … WITH NORECOVERY: Run the command, specifying a clear path and filename.
• Verify Backup File: Ensure the .trn file was created successfully.
• Confirm Database State: Check that the source database is now in the RESTORING state.
• Proceed with Restore: Use the tail-log backup as the final restore step on the target server or for recovery.
• Set database to MULTI_USER: If you’ve placed the source database in SINGLE_USER mode just prior to performing the tail-log backup, the restored database on the target will be in SINGLE_USER mode as well. To allow connections, be sure to switch it to MULTI_USER.

In conclusion

While often overlooked in basic backup discussions, the tail-log backup is a vital tool in the SQL Server DBA’s toolkit. It provides the critical ability to capture the very last transactions before a database restore or migration, minimizing data loss and ensuring the most up-to-date recovery possible. Understanding when and how to use it is key to robust data protection and seamless database migrations.

Want to work with The SERO Group?

Want to learn more about how The SERO Group helps organizations take the guesswork out of managing their SQL Servers? Schedule a no-obligation discovery call with us to get started.

The post Why Tail-Log Backups Matter for SQL Server Recovery and Migration appeared first on The SERO Group.

Fortunately, if you’re using SQL Server Enterprise and require quicker access to only a subset of data, piecemeal restores may be your answer to faster recovery.

Piecemeal restores are available in both Standard and Enterprise editions. However, with Enterprise, you can perform these online.

In this post, I’ll cover:

• What is an online piecemeal restore?
• Requirements for performing an online piecemeal restore.
• Best practices for online piecemeal restores.

What is an online piecemeal restore?

An online piecemeal restore is a multi-stage process that allows partial restoration of a database while keeping the primary filegroup available for access. This is particularly useful for:

• Minimizing downtime: The primary filegroup remains online while secondary filegroups are restored.
• High Availability: Users can continue working with critical data while less frequently accessed data is recovered.
• Efficient Disaster Recovery: Restoring large databases in phases instead of performing a full database restore.

Requirements for performing an online piecemeal restore

Before you can perform an online piecemeal restore, ensure that

1. Enterprise edition of SQL Server: Online piecemeal restores are only supported in SQL Server Enterprise edition.
2. Database uses multiple filegroups: The database must utilize multiple filegroups.
3. Database uses full recovery model: Ensure the database is using the full recovery model.
4. Primary filegroup remains online: The primary filegroup must be online before restoring secondary filegroups.
5. Log backups are available: Transaction logs must cover the period between the full backup and the latest (differential or filegroup) backup.
6. No cross-filegroup constraints: Foreign key constraints referencing objects in different filegroups can block an online restore. However, FKs that reference objects in the primary filegroup will not cause an issue (the primary filegroup must be restored first and online before all secondary filegroups can be restored).

Step-by-step guide to performing an online piecemeal restore

Step 1: Verify database filegroups

Before initiating a restore, check the current filegroups in the database:

select name, type_desc from sys.filegroups;

Step 2: Restore the primary filegroup

Restore the primary filegroup and bring the database online. If you are restoring additional differential or log backups, include “NORECOVERY” in your restore statement. However, in this example, we’re only restoring the FULL backup and then bringing the database online. Take note of the “PARTIAL” keyword:

RESTORE DATABASE MyDatabase FILEGROUP = 'PRIMARY' FROM DISK = '\\myshare\mydatabase\full\MyDatabase_FULL.bak' 
WITH PARTIAL, STATS = 10;

Step 3: Restore additional filegroups

Each additional filegroup must be restored. If a query attempts to access an object that is in the secondary filegroup, an error will be returned stating the filegroup is offline.

RESTORE DATABASE MyDatabase FILEGROUP = 'MyFileGroup' FROM DISK = '\\myshare\mydatabase\full\MyDatabase_FULL.bak'
WITH STATS = 10;

At this point, the database is fully available for use.

Best practices for online piecemeal restores

• To support online piecemeal restores, Enterprise edition must be used.
• Ensure no foreign keys or indexed views span multiple filegroups that may block restores.

Want to work with The SERO Group?

Want to learn more about how The SERO Group helps organizations take the guesswork out of managing their SQL Servers? Schedule a no-obligation discovery call with us to get started.

The post SQL Server Piecemeal Restores: Minimize Downtime & Speed Up Recovery appeared first on The SERO Group.

I’ve encountered managers, systems administrators, and developers who’ve wondered this. Simply put, data reads are much faster in RAM than from disks. RAM latency is typically measured in nanoseconds. By contrast, disk latency is measured in milliseconds.

It’s not uncommon to find dedicated SQL Server machines with 90% of memory allocated to the sqlservr.exe process (another reason to use dedicated SQL Server machines, but that’s for another post).

Let’s take a high-level look at how SQL Server uses memory so the next time you’re asked why memory usage is so high, you’ll be able to respond and explain why this is typically ok.

Buffer Pool (Cache)

This is the largest memory area used for caching data and index pages. SQL Server keeps data in memory to minimize expensive disk I/O operations. Many things can affect how long pages are kept in the cache, but we won’t cover those here. When you adjust SQL Server’s max memory setting, you’re telling SQL Server how much memory it can use for the Buffer Pool (this affects other types of memory allocations as well, but the Buffer Pool is the largest consumer).

If you’ve adjusted max memory but still see that the sqlservr.exe process is consuming more, it’s because SQL Server has additional memory requirements unaffected by this setting. See “SQL Server might commit memory over the max memory setting” for some reasons why you may be experiencing this.

Plan Cache

The plan cache stores execution plans to avoid the cost of recompiling queries. You see, when SQL Server compiles a query, it must perform several steps:

1. Parse the query.
2. Bind it by resolving objects (tables, columns, etc) referenced in the query to ensure they exist. Validate data types, perform implicit conversions (try to avoid these if possible), and ensure logical operators in the query are bound to database objects to ensure it can actually execute.
3. Next, the optimizer takes the parsed query and attempts to generate the most efficient execution plan (the keyword here is attempts). This is a critical step.
4. Next, the plan is finalized and converted into a format SQL Server can execute.
5. Then, the plan is stored in the plan cache for future reuse, provided the same query is executed.

Just like with the Buffer Pool, various things can affect plan reuse or force a query to be recompiled. Again, I won’t go too far into that today.

The max memory setting influences the plan cache size and is managed dynamically within this allocation. For a more in-depth understanding of SQL Server’s plan cache limits and memory management, see Erin Stellato’s blog post on SQLskills.

Worker Threads and Connections

SQL Server needs memory to manage worker threads and connections. These are critical components of its query execution engine.

Worker threads are responsible for executing tasks in SQL Server (running queries, handling requests, and performing background tasks). Each thread needs a certain amount of memory to function. These can be adjusted through the max worker threads configuration setting. I can’t recall a time I’ve ever needed to change this setting. If you find yourself thinking you need to change the default, check out Brent Ozar’s post here.

Each client’s connection to SQL Server requires memory to handle communication and processing.

• Connection Context – Session settings, Query and command states, etc.

If you want to take a deep dive into threads, review Microsoft’s Thread and Tasks Architecture Guide.

Memory for Internal Processes

SQL Server performs various operations during query execution and other activities. It needs memory to do that.

• Locking
• Sorting
• Hash Joins
• Temporary Objects (that don’t spill over to tempdb)

In-memory OLTP (Memory-Optimized Tables)

This feature was introduced in SQL Server 2014 and designed for high-performance transactions. It can consume large amounts of memory; you should plan carefully before using this feature. See here for more information on managing memory allocated to these objects using SQL Server’s Resource Governor.

In Conclusion

High memory usage in SQL Server is not a problem—it’s a feature that improves performance by reducing reliance on slower disk operations.

Ready to Optimize Your SQL Server?

Discover how The SERO Group can take the guesswork out of SQL Server management. Schedule a no-obligation discovery call today!

The post Understanding SQL Server’s High Memory Usage: What You Need to Know appeared first on The SERO Group.

Multi-Attach EBS volumes in AWS enable organizations to use Windows Server Failover Clusters with shared storage. Attaching a single EBS (Elastic Block Store) to multiple EC2 instances simultaneously makes them ideal for SQL Server FCIs.

What are Multi-Attach EBS Volumes?

These EBS volumes are single Elastic Block Store volumes that can be attached to multiple EC2 instances. The feature is available for io1 and io2 Block Express volumes. However, Windows instances only support io2 Block Express volumes. Multi-Attach facilitates high availability and seamless failover for SQL Server FCIs (similar to how they operate on-premises).

Prerequisites

Before setting up these volumes for SQL Server FCIs, ensure the following prerequisites are met:

1. EC2 Instances in the same Availability Zone

All EC2 instances that will attach to the EBS volume must reside within the same AZ. This ensures low latency and high throughput between the instances and the EBS volume. This is critical for maintaining performance and reliability in a failover scenario.

2. Supported EBS Volume Types

While both io1 and io2 Block Express volumes support the Multi-Attach option, Windows instances only support io2 Block Express volumes. This limitation is due to the enhanced performance and features that io2 Block Express provides.

3. Regional Availability of io2 Block Express Volumes

Verify that io2 Block Express Volumes are available in your AWS region. Not all regions may support this volume type, so review the Amazon EBS Provisioned IOPS SSD volumes user guide to confirm availability.

4. Additional Networking and VPC Configurations

While this post focuses on EBS Multi-Attach, ensure that your networking and VPC configurations meet the requirements for setting up a Windows Server Failover Cluster. See this link to review the full list of prerequisites for WSFC.

io2 Block Express Volumes

These volumes offer the highest-performance block storage. Key features include:

• Capacity: Ranges from 4 GB to 64 TB, accommodating various workloads.
• Performance:
  • IOPS: Up to 256,000 IOPS per volume.
  • Throughput: Maximum throughput of 4 GB/s per volume

These high-performance metrics make io2 Block Express volumes ideal for demanding SQL Server workloads, especially those requiring rapid data access and high transaction rates.

Compatibility

io2 Block Express volumes are available on all Nitro-based EC2 instances. The AWS Nitro system is a collection of hardware and software components designed to provide enhanced security, performance, and flexibility for EC2 instances.

Enabling Multi-Attach for EBS Volumes

By default, this option is disabled but simple to enable. See this link for instructions on using the AWS Console or Command-Line.

Want to work with The SERO Group?

Want to learn more about how The SERO Group helps organizations take the guesswork out of managing their SQL Servers? Schedule a no-obligation discovery call with us to get started.

The post How to Build SQL Server FCIs in AWS with Multi-Attach EBS Volumes appeared first on The SERO Group.

Monitoring the growth of your SQL Server transaction log files is crucial for maintaining database performance and ensuring system reliability. Unchecked transaction log growth can lead to disk space issues and system outages. Let’s take a look at some of the causes of transaction log file growth, why it’s important to monitor, and how you can use the sp_whoisactive procedure to keep an eye on it.

What Causes the Transaction Log File to Grow

The transaction log file records all transactions and database modifications made by each transaction. Several factors can contribute to its growth:

• Long-Running Transactions: Transactions that take a long time to complete prevent the log from truncating because SQL Server needs to maintain the log records until the transaction is committed.
• Recovery Models:
  • Full Recovery Model: In this model, the transaction log will not truncate until a log backup occurs. If log backups are not taken regularly, the log file will grow indefinitely.
  • Bulk-Logged Recovery Model: Similar to the full recovery model, it is used for bulk operations. Log growth can still be an issue if backups aren’t scheduled appropriately.
  • Simple Recovery Model: Although SQL Server automatically truncates the log in this recovery model, you must still watch for transactions that are making large changes all at once. For instance, if you update millions of rows in one transaction, this could be enough to cause the log file to grow.
• Lack of Log Backups: Without regular log backups in the full or bulk-logged recovery models, the log cannot truncate, leading to continuous growth.
• High Transaction Volume: A sudden spike in transaction activity can cause the log file to grow rapidly to accommodate the increased volume.
• Replication and Availability Groups: Features like availability groups and replication can delay log truncation until the data is sent to the replicas or replicated databases. Seeing the log file continue to grow even after taking a log backup? Check the log_reuse_wait_desc column in sys.databases. It’ll indicate why.
• Open Transactions: Uncommitted transactions hold on to log space because SQL Server needs the log records to roll backup if necessary. Who hasn’t started their session with “begin transaction” and gone home for the weekend (forgetting the commit)?

Why it Matters

The impact can be massive, from crashing applications to filling up disk space and affecting all other databases on the same drive.

• Disk Space Consumption: Uncontrolled log growth can consume all available disk space, leading to application failures and downtime.
• Performance Degradation: Large log files can slow down database operations, including backups and recovery processes, affecting overall system performance.
• Recovery Time Objectives (RTOs): In disaster recovery scenarios, large transaction logs can increase the time it takes to restore databases, impacting business continuity.
• Maintenance Challenges: Managing and maintaining oversized log files can be cumbersome, requiring more time for routine operations like backups.
• Risk of Data Loss: If the disk runs out of space due to log growth, new transactions cannot be logged, leading to potential data loss or corruption.

How to monitor it using sp_WhoIsActive

sp_whoisactive is a powerful stored procedure developed by Adam Machanic that provides a comprehensive view of the current activity on your SQL Server. It can be used to monitor transaction log growth.

Here’s how you can use it:

Installation

First, download the script from https://github.com/amachanic/sp_whoisactive and run it on your instance. I usually place it in a database named DBA vs. master.

Monitoring Active Sessions

Run the procedure to see all active sessions:

USE DBA
GO
EXEC sp_whoisactive

Identifying Long-Running Transactions

To focus on transactions that might be causing log growth, look for sessions with high values in the tran_log_writes column, which shows the number of log records written by the session. You’ll need to set the @get_transaction_info to 1 to include transaction details.

EXEC sp_whoisactive @get_transaction_info = 1

This will add columns related to transaction duration and log usage.

Ordering by log writes

If you’re interested in sessions consuming the most log space, order the results accordingly:

EXEC sp_whoisactive @get_transaction_info = 1, @sort_order = '[tran_log_writes] DESC'

Automating Monitoring

Schedule sp_whoisactive to run at regular intervals and log the output to a table for historical analysis or to send an alert if certain thresholds are met. This can help you identify patterns over time (i.e., Mike usually starts a new session every Friday evening and leaves for the weekend.). The following code will create a table, dbo.WhoIsActiveLog, with the appropriate schema. The second statement will then execute sp_whoisactve and store the results within this table.

--create the dbo.WhoIsActiveLog table
DECLARE @schemaDefinition VARCHAR(MAX)
EXEC sp_WhoIsActive 
    @find_block_leaders = 1, 
	@get_transaction_info = 1,
	@get_locks = 1,
	@return_schema = 1,
	@schema = @schemaDefinition OUTPUT

SELECT @schemaDefinition =  REPLACE(@schemaDefinition,'<table_name>','dbo.WhoIsActiveLog')
EXEC (@schemaDefinition)
GO
EXEC sp_WhoIsActive 
    @find_block_leaders = 1, 
	@get_transaction_info = 1,
	@get_locks = 1,
	@destination_table = 'dbo.WhoIsActiveLog'

Once you identify sessions causing excessive log growth, you can investigate the underlying queries or processes:

• Optimize Queries: Rewrite inefficient queries to reduce transaction time.
• Adjust Recovery Models: If appropriate, and point-in-time recovery isn’t required, consider switching to the simple recovery model. However, this won’t help if it’s a long running transaction making many changes simultaneously.
• Schedule Regular Backups: Ensure that log backups are taken frequently to prevent log file bloat.
• Kill Problematic Sessions: As a last resort, terminate sessions that are causing issues, but be cautious of potential data loss.

Want to work with The SERO Group?

Want to learn more about how The SERO Group helps organizations take the guesswork out of managing their SQL Servers? Schedule a no-obligation discovery call with us to get started.

The post Prevent SQL Server Outages by Monitoring Transaction Log Growth appeared first on The SERO Group.

Although Basic AGs were introduced before SQL Server 2022, we’ll focus strictly on the latest version.

Let’s review the requirements and limitations.

Requirements and Limitations

SQL Server Edition and Version:

• Standard Edition: Both the primary and secondary replicas must be running SQL Server 2022 Standard Edition. Basic Availability Groups are unavailable in the Express, Web, or Enterprise editions.
• Same Version: Ideally, all replicas should be running the same build of SQL Server 2022.

Verify the version by running SELECT @@VERSION. In the screenshot below, you can see which update has been applied and the SQL Server version and edition.

Windows Server Failover Clustering (WSFC):

• Cluster Configuration: A Windows Server Failover Cluster must be configured. All nodes participating in the Basic Availability Group must be members of the same WSFC.
• Operating System: The servers must be running a Windows Server version that supports failover clustering (e.g., Windows Server 2019 or 2022 recommended).
• Domain Membership: All nodes must be members of the same Active Directory domain or trusted domains.

In our example, we have a two-node WSFC cluster using Windows Server 2022 Standard Edition.

Network Requirements:

• Connectivity: Reliable network connections between the replicas are essential.
• Name Resolution: A proper DNS setup for name resolution between nodes is preferred.
• Static IPs: It’s highly recommended all replicas utilize static IP addresses vs. DHCP-assigned IPs.

Database Requirements:

• Full Recovery Model: The database to be included in the Basic Availability Group must use the Full recovery model.
• Single Database Support: Basic Availability Groups support only one user database per group.
• Initialization: The database must be initialized on the secondary replica using a full backup and transaction log backups. Automatic seeding can be used as well.

Replica Limitations:

• Number of Replicas: Only two replicas are supported. One primary and one secondary.
• Secondary Replica Limitations
  • Non-Readable: The secondary replica is not accessible for read operations.
  • No backups: You cannot perform backups on the secondary replica.
  • Synchronous Commit: To support automatic failover, Availability Mode must be set to synchronous commit. This is also a requirement for standard Availability Groups.

Licensing:

• Proper Licensing: Ensure that all SQL Server instances are properly licensed under Microsoft’s licensing terms for high-availability setups. See Microsoft’s licensing documentation for full details (SQL Server 2022—Pricing | Microsoft).

Permissions and Roles:

• Service Accounts: To reduce complexity, it’s recommended to use the same SQL Server service account on both replicas. Consider using group managed service accounts for further simplification.
• Firewall Settings: Configure Windows Firewall to allow communication between the replicas on the required ports (default is TCP port 5022).

Endpoint Configuration:

• Database Mirroring Endpoint: This must be configured on each replica for data synchronization. If using the wizard in SSMS, the endpoint is created and configured.

Quorum Configuration:

• Proper quorum configuration is essential to maintain cluster health and support automatic failover. This is done within the WSFC.

Summary of Limitations

• Single Database Limitation: Only one database can be included in a Basic Availability Group.
• Limited to Two Replicas: Only one primary and one secondary replica are allowed.
• No Read-Scale Capabilities: Unlike Availability Groups in Enterprise Edition, Basic Availability Groups do not support readable secondary replicas.
• No Backup Offloading: Backups cannot be performed on the secondary replica.

Conclusion

Basic Availability Groups can fill the gap if your environment doesn’t support a WSFC with the shared disk. However, due to their limitation of supporting only one database, ongoing management of multiple Basic AGs may not be ideal. For example, I’d still use a failover clustered instance if any of the following were true:

• SQL Server Standard Edition is used.
• Not spanning multiple data centers.
• The environment allows for the use of shared disks between machines.

Want to work with The SERO Group?

Want to learn more about how The SERO Group helps organizations take the guesswork out of managing their SQL Servers? Schedule a no-obligation discovery call with us to get started.

The post Basic Availability Groups: Affordable High Availability with Key Limitations appeared first on The SERO Group.

What are contained Availability Groups?

Contained AGs enhance the traditional Availability Groups (AGs) introduced in SQL Server 2012. CAGs include user databases and the associated server objects, such as logins, permissions, and SQL Server Agent jobs. Contained AGs automatically sync these objects across replicas. This saves the DBA from having to manually make these changes.

Key benefits of contained AGs

Ease of Management: Contained AGs bundle both databases and associated instance-level objects, simplifying management. This especially simplifies managing complex environments in which an application depends on resources outside of the database replica. Before CAGs, the synchronization of these objects required home-grown solutions using T-SQL and/or PowerShell.

Seamless Failover: With traditional AGs, failover could sometimes result in missing instance-level objects, leading to broken applications. CAGs ensure that all necessary objects are included within the group, providing a seamless failover experience.

Getting started

To get started, you’ll need:

• A Windows Server Failover Cluster (preferably Windows Server 2022) with at least two nodes.
• SQL Server 2022 Enterprise Edition. Developer Edition works for testing purposes.

Additionally, the following resources can be used as a guide for building your contained AG:

• SQL Server Contained Availability Groups Configuration (mssqltips.com)
• What is a contained availability group? – SQL Server Always On | Microsoft Learn

Want to work with The SERO Group?

Learn how The SERO Group can simplify SQL Server management for your organization. Schedule a no-obligation discovery call with us to get started.

The post Streamline SQL Server Management with Contained Availability Groups appeared first on The SERO Group.

Involve Stakeholders Early in the Process of Building a SQL Server Availability Group

Before jumping into building your Availability Group, it’s crucial to identify key players in this process. Understanding these stakeholders’ roles and responsibilities will help you to plan, execute, and maintain the AG.

Build Your Team: Key Roles in SQL Server Availability Group Setup

Here’s a list of common roles in this process. Your specific list will depend on your organization’s structure.

Database Administrator (DBA)

The DBA is the primary player in setting up and managing the SQL Server AG. They’re usually responsible for:

• Designing the AG architecture.
• Regularly testing failover scenarios.
• Configuring and maintaining the AG.
• Ensuring data consistency and availability.
• Implementing backup and restore strategies.
• Monitoring performance and troubleshooting issues.
• Monitoring synchronization latency between all replicas.
• Planning the AG setup, including replicas, failover modes, and quorum configurations.
• Assessing the current database environment (including the recovery models of each database to be added to the AG).

Network Administrator

The Network Administrator ensures that the network infrastructure supports the high availability and redundancy required for the SQL Server AG. Common responsibilities include:

• Configuring and managing network components.
• Making sure network redundancy is built-in to the solution (replicas having multiple NICs connected to different network switches).
• Ensuring low-latency and high-bandwidth connections between replicas.
• Setting up network security measures, such as firewalls and VLANs.
• Monitoring network performance and troubleshooting connectivity issues.
• Static IP assignment for each replica, AG listener, and Cluster Network Object (CNO).

System Administrator

System Administrators manage the underlying servers and operating systems that host the SQL Server instances involved in the AG. Responsibilities include:

• Preparing and configuring the Windows Server environment (virtual or physical).
• Installing necessary software and updates.
• Ensuring proper server performance and resource allocation.
• Implementing security policies and hardening the server configurations.
• Building the Windows Server Failover Cluster.
• Partitioning and formatting disks according to the correct specifications (can vary depending on the storage vendor).

Storage Administrator

Storage administrators manage the storage infrastructure that the SQL Server instances rely on. Ensuring that each disk(s) presented to the instances are:

• Provisioned and configured according to performance and capacity requirements.
• Highly available and redundant.
• Monitored for storage performance and health.
• Implemented in a way that meets the requirements of any established disaster recovery plan.

Application Developers

Developers are crucial for ensuring applications are correctly configured to connect to the AG and handle failover scenarios. If the application was created by a vendor, documentation must be reviewed to ensure AG support is available. Their responsibilities include:

• Modifying connection strings to use the AG listener.
• Ensuring the application logic can handle failovers without data loss or significant downtime.
• Testing application behavior in failover and recovery scenarios.

Business Stakeholders

This group defines the availability requirements and success criteria for the SQL Server AG. Their responsibilities include:

• Setting Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO).
• Prioritizing databases and applications that require high availability.
• Allocating budget and resources for the AG project.
• Communicating the business impact and benefits of the AG to the organization.

Many AG implementations fail not because of technical limitations, but because expectations were not set. Ensure expectations for RTO and RPO are over-communicated, documented, and agreed upon by all groups involved.

Security Team

This team ensures that the AG implementation complies with organizational security policies and regulatory requirements. It’s normal for there to be some overlap between the security team and DBA(s). Security’s responsibilities may include:

• Reviewing and approving security configurations.
• Implementing encryption for data in transit and at rest (shared responsibility with the DBA if required).
• Monitoring and auditing access to the AG.
• Ensuring compliance with industry standards and regulations.

Project Manager

The PM coordinates the overall AG implementation project, ensuring that timelines are met and resources are allocated efficiently. They keep everyone on track. Responsibilities include:

• Creating a project plan with clear milestones and deadlines.
• Facilitating communication between all stakeholders.
• Managing risks and issues that arise during the project.
• Ensuring that all deliverables meet quality standards.

Conclusion

These roles may vary across organizations. A specific team may also play multiple roles. However, identifying and involving these key players early in the planning phase of your AG project is critical for success. Each stakeholder brings unique expertise and insights that are vital for building a robust, efficient, and reliable availability group. You can ensure a seamless implementation that meets both technical and business requirements by fostering collaboration and clear communication among all roles.

Want to work with The SERO Group?

Want to learn more about how SERO Group helps organizations take the guesswork out of managing their SQL Servers? Schedule a no-obligation discovery call with us to get started.

The post Building an Availability Group? Involve Key Players First. appeared first on The SERO Group.

But why now? They’ve been in business for years and are just now interested in HA/DR. What changed?

Reasons vary, but the following are very common:

• A new cyber insurance policy requiring a HA/DR or business continuity plan.
• A need for HA/DR capability in order to create or develop a new revenue stream.
• General curiosity due to SQL Server marketing materials.
• A new client or business group who has asked about HA/DR.
• Concerns raised during a recent audit.
• Recent outage that impacted revenue.

SQL Server has several options to choose from when it comes to HA/DR. These include:

• Log Shipping: No automatic failover capabilities. Available on SQL Server Standard and Enterprise editions.
• Database Mirroring: Deprecated, don’t use it in new environments.
• Failover Cluster Instances: Available on SQL Server Standard and Enterprise editions.
  • What’s the Difference in SQL Server FCIs and AGs?
• Availability Groups

Since you’re here to learn more about availability groups, I’ll introduce you to the two main types.

Availability Groups

Availability Groups were introduced in SQL Server 2012 Enterprise Edition. They provide advanced high availability and disaster recovery capabilities.

• Edition Support: Available in SQL Server 2012 Enterprise Edition and later. FYI: Official support for SQL Server 2012 ended in 2022.
• Number of Replicas: SQL Server 2022 specs – One primary and up to 8 secondary replicas (under asynchronous-commit mode or up to 5 can run under synchronous-commit mode).
• Supports readable secondaries: Yes
• Failover: Supports automatic failover for synchronous-commit replicas and manual failover for asynchronous-commit replicas.
• Use Cases:
  • Ideal for mission-critical applications requiring high availability and disaster recovery across multiple databases.
  • Suitable for enterprises needing advanced features like read-only secondary replicas for offloading read workloads.

Requirements:

• Same OS version on all nodes.
• Proper network infrastructure to support data replication with minimal latency.
• Windows Server Failover Clustering (WSFC). There are some exceptions to this that I’ll cover in a later post.
• SQL Server 2019 Enterprise Edition or later. Available with earlier versions, but those are out of mainstream support. I wouldn’t want you building a new AG on unsupported versions. In fact, that would almost certainly raise flags in other areas of your cyber insurance policy.

Basic Availability Groups

Up until SQL Server 2016, AGs were only available in Enterprise Edition. However, in 2016, Microsoft introduced us to Basic Availability Groups. Basic AGs provide an introduction to high availability and disaster recovery solutions for a single database.

• Edition Support: Available in SQL Server 2016 Standard Edition and later.
• Number of Replicas: Supports two replicas (one primary and one secondary).
• Supports readable secondaries: No. See the limitations here.
• Failover: Supports automatic failover, but only in a single-database environment.
• Use Cases:
  • Ideal for small businesses or environments where budget constraints prevent the use of Enterprise Edition.
  • Suitable for applications that require minimal HA/DR capabilities.

Requirements:

• Same OS version on both nodes.
• SQL Server 2016 Standard Edition or later.
• Windows Server Failover Clustering (WSFC)
• Proper network infrastructure to support data replication with minimal latency.

Which should you choose?

Getting AGs right takes practice. When done wrong, performance and availability can both suffer. Budgets can be destroyed by licenses. Understanding your options, including their requirements and differences, can help you choose the right solution for your environment. Whether you’re driven by audits, insurance policies, or those uncomfortable “could not open a connection to SQL Server” errors, investing in HA/DR solutions is crucial for maintaining business continuity and minimizing downtime.

So, which would I choose? If I had the option to use shared storage, I’d opt for building a failover clustered instance instead of using a Basic AG. FCIs support failover of multiple databases and can be used with Standard Edition. However, if shared storage isn’t an option and you need automatic failover capabilities for a single database or if you’re working in a hybrid environment (Azure and on-premises), Basic AGs may be a good choice for you.

Have questions about HA/DR? Here are two ways we can help.

1. Schedule a call with us to connect with our consultants. We’ve helped numerous clients just like you meet their uptime objectives.
2. Consider SEROShield, our DBA team as a service. Not only will we help build the AG, but we’ll also support it. It’ll be like driving around in a new car (you know, the one with the new car smell) that you know is covered by the best possible insurance.

Next in this series

There are a few lesser-known availability group types that we didn’t discuss today but will in upcoming posts: Contained AGs, Distributed AGs, and Domain-independent AGs.

Follow us on LinkedIn to get notified when we release new blog posts.

The post Availability Groups: What’s Right for Your Business? appeared first on The SERO Group.